Load balancing method and device, electronic equipment and storage medium

Abstract

The invention relates to a load balancing method and device, electronic equipment and a storage medium. The method comprises the steps that a received data writing request is responded, a plurality of nodes in the distributed storage system are obtained, and the data writing request comprises target data and a target data type; determining a plurality of write-in nodes from the plurality of nodes according to the target data type; collecting node load information of a plurality of write-in nodes, and calculating a comprehensive load score of the corresponding write-in node according to the node load information; obtaining a data size corresponding to a target data type in the plurality of write-in nodes, and determining a target write-in node from the plurality of write-in nodes according to the comprehensive load score and the data size corresponding to the target data type; and writing the target data into the target writing node to respond to the data writing request. By adopting the method, more accurate and efficient load balancing can be realized, and the overall performance and the resource utilization rate of the system are improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a load balancing method, device, electronic device and storage medium. Background Art

[0002] In the era of big data, enterprise data is growing rapidly and showing a trend of massive data. Currently, enterprises usually use distributed storage systems for data storage. Distributed storage systems are composed of multiple nodes, and each node can store a part of the data. When data is written into a distributed storage system, load balancing or consistent hashing is usually used to calculate which node the data should be stored in.

[0003] Traditional load balancing methods are often based on a single load indicator, such as processor usage, memory occupancy, or disk input and output rate, which is difficult to fully reflect the overall load status of the system. In addition, due to the continuous increase in data types, each node stores data of multiple data types, resulting in a large number of data distribution operations required for subsequent data reading, resulting in a decrease in system performance.

[0004] Therefore, there is an urgent need to propose a load balancing method, device, electronic device and storage medium that can achieve more accurate and efficient load balancing and improve the overall system performance and resource utilization. Summary of the invention

[0005] Based on this, it is necessary to provide a load balancing method, device, electronic device and storage medium that can achieve more accurate and efficient load balancing and improve the overall system performance and resource utilization in response to the above technical problems.

[0006] In a first aspect, a load balancing method is provided, which is applied to a distributed storage system, wherein the distributed storage system includes a plurality of nodes, and the method includes:

[0007] In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type;

[0008] Determine a plurality of write nodes from the plurality of nodes according to the target data type;

[0009] Collect node load information of multiple write nodes, and calculate the comprehensive load score of the corresponding write node according to the node load information;

[0010] Obtaining data sizes corresponding to target data types from multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data sizes corresponding to the target data types;

[0011] Writes the target data to the target write node in response to the data write request.

[0012] In one embodiment, determining a plurality of write nodes from a plurality of nodes according to a target data type includes:

[0013] Obtaining data type distribution in a plurality of nodes, wherein the data type distribution represents a plurality of data type types included in the corresponding nodes;

[0014] Match the data type distribution of multiple nodes with the target data type to obtain a type matching result;

[0015] In response to the type matching result indicating that the corresponding node does not contain the data type kind of the target data type, determining the corresponding node as another node;

[0016] In response to the type matching result indicating that the corresponding node contains the data type kind of the target data type, the corresponding node is determined as a write node.

[0017] In one embodiment, the node load information includes multiple load information of the corresponding write node, and the comprehensive load score of the corresponding write node is calculated according to the node load information, including:

[0018] Obtaining load assessment criteria, wherein the load assessment criteria include multiple load information scoring criteria;

[0019] Matching a plurality of load information with corresponding load information scoring criteria to obtain a plurality of load information scores;

[0020] The comprehensive load score of the corresponding write node is calculated based on the multiple load information scores.

[0021] In one embodiment, multiple load information is matched with corresponding load information scoring criteria to obtain multiple load information scores, including:

[0022] Matching the processor usage with the processor usage scoring standard to obtain a processor usage matching result, wherein the processor usage matching result includes a processor usage score;

[0023] Matching the memory usage with the memory usage scoring standard to obtain a memory usage matching result, wherein the memory usage matching result includes a memory usage score;

[0024] Matching the actual operation status with the actual operation status scoring standard to obtain an actual operation status matching result, wherein the actual operation status matching result includes an actual operation status score;

[0025] The business requirement satisfaction is matched with the business requirement satisfaction scoring standard to obtain a business requirement satisfaction matching result, wherein the business requirement satisfaction matching result includes a business requirement satisfaction score.

[0026] In one embodiment, the comprehensive load calculation rule includes a processor usage weight, a memory occupancy weight, an actual operation weight, and a business demand satisfaction weight, and the comprehensive load score of the corresponding write node is calculated according to the multiple load information scores, including:

[0027] Multiplying the processor usage score by the processor usage weight to obtain a first load score;

[0028] Multiply the memory usage score by the memory usage weight to obtain a second load score;

[0029] Multiply the actual operation condition score by the actual operation condition weight to obtain a third load score;

[0030] Multiply the business demand satisfaction score by the business demand satisfaction weight to obtain a fourth load score;

[0031] The first load score, the second load score, the third load score, and the fourth load score are added together to obtain a comprehensive load score of the corresponding write node.

[0032] In one embodiment, obtaining data sizes corresponding to target data types in multiple write nodes, and determining a target write node from the multiple write nodes according to a comprehensive load score and the data sizes corresponding to the target data types, includes:

[0033] Obtaining a first set threshold, and comparing the comprehensive load scores of the plurality of write nodes with the first set threshold to obtain a first comparison result;

[0034] Screening out a plurality of first write nodes according to the first comparison result, wherein the comprehensive load score of the first write node is less than or equal to a first set threshold;

[0035] Obtaining a second set threshold, and comparing the data size corresponding to the target data type in the plurality of first write nodes with the second set threshold to obtain a second comparison result;

[0036] Filter out a plurality of second write nodes according to the second comparison result, wherein the data size corresponding to the target data type of the second write node is less than or equal to the second set threshold;

[0037] Sort the plurality of second write nodes from large to small according to the data sizes corresponding to the target data type, to obtain a data size sorting result;

[0038] According to the data size sorting result, the second write node ranked first is used as the target write node.

[0039] In one embodiment, writing the target data into the target write node, the method further includes:

[0040] Determine whether the distributed storage system meets the load balancing condition, wherein the load balancing condition is that the difference in the comprehensive load scores of the distributed storage system is less than a third set threshold, and the difference in the comprehensive load scores is the difference between the maximum comprehensive load score and the minimum comprehensive load score in the distributed storage system;

[0041] In response to the distributed storage system not satisfying the load balancing condition, determining whether a node with a high comprehensive load score and a node with a low comprehensive load score store data of the same data type;

[0042] In response to the node with a high comprehensive load score and the node with a low comprehensive load score storing data of the same data type, transferring the data of the data type with the largest data size on the node with a high comprehensive load score to the node with a low comprehensive load score;

[0043] In response to the fact that the node with a high comprehensive load score and the node with a low comprehensive load score do not store data of the same data type, the smallest file on the node with a high comprehensive load score is transferred to the node with a low comprehensive load score.

[0044] In a second aspect, a load balancing device is provided, the device comprising:

[0045] A first acquisition module, the first acquisition module is used to acquire multiple nodes in the distributed storage system in response to receiving a data write request, wherein the data write request includes target data and a target data type;

[0046] A first determination module, the first determination module is used to determine a plurality of write nodes from a plurality of nodes according to a target data type;

[0047] A collection module, the collection module is used to collect node load information of multiple write nodes;

[0048] A calculation module, which is used to calculate the comprehensive load score of the corresponding write node according to the node load information;

[0049] A second acquisition module, the second acquisition module is used to acquire the data size corresponding to the target data type in multiple write nodes;

[0050] A second determination module, the second determination module is used to determine a target write node from multiple write nodes according to the comprehensive load score and the data size corresponding to the target data type;

[0051] The write module is used to write target data to a target write node in response to a data write request.

[0052] In a third aspect, an electronic device is provided, comprising one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of any one of the methods in the first aspect described above.

[0053] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods in the first aspect are performed.

[0054] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0055] The load balancing method, device, electronic device and storage medium determine multiple write nodes according to the target data type, and determine the target write node from the multiple write nodes to reduce the data distribution operation when reading data subsequently, thereby improving system performance. According to the data size and comprehensive load score corresponding to the target data type in the multiple write nodes, the target write node is determined from the multiple write nodes, achieving more accurate and efficient load balancing, and improving the resource utilization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a load balancing method in one embodiment;

[0057] Figure 2 It is a schematic diagram of the load balancing method in a distributed storage system;

[0058] Figure 3 is a structural block diagram of a load balancing device in one embodiment;

[0059] Figure 4 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] Embodiment 1

[0062] In one embodiment, Figure 1 As shown, a load balancing method is provided, which is applied to Figure 2The distributed storage system shown in the figure includes multiple distributed storage clusters (nodes), a data acquisition module, a load balancing module, a load evaluation module, and a health check and fault recovery module. The method includes:

[0063] In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type;

[0064] Determine a plurality of write nodes from the plurality of nodes according to the target data type;

[0065] Collect node load information of multiple write nodes, and calculate the comprehensive load score of the corresponding write node according to the node load information;

[0066] Obtaining data sizes corresponding to target data types from multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data sizes corresponding to the target data types;

[0067] Writes the target data to the target write node in response to the data write request.

[0068] Specifically, multiple write nodes are determined according to the target data type, and the target write node is determined from the multiple write nodes to reduce the data distribution operation when reading data subsequently, thereby improving system performance. According to the data size and comprehensive load score corresponding to the target data type in the multiple write nodes, the target write node is determined from the multiple write nodes, achieving more accurate and efficient load balancing and improving the resource utilization of the system.

[0069] In one embodiment, determining a plurality of write nodes from a plurality of nodes according to a target data type includes:

[0070] Obtaining data type distribution in a plurality of nodes, wherein the data type distribution represents a plurality of data type types included in the corresponding nodes;

[0071] Match the data type distribution of multiple nodes with the target data type to obtain a type matching result;

[0072] In response to the type matching result indicating that the corresponding node does not contain the data type kind of the target data type, determining the corresponding node as another node;

[0073] In response to the type matching result indicating that the corresponding node contains the data type kind of the target data type, the corresponding node is determined as a write node.

[0074] Specifically, using nodes storing data of the target data type as write nodes reduces data distribution operations during subsequent data reading and improves system performance.

[0075] In a specific embodiment, data types can be mainly divided into the following three categories: 1) Unstructured data: Unstructured data refers to data that has no clear association relationship between them, and the data format is diverse, such as text, pictures, audio, video, etc. This type of data is often stored in the form of file objects in a distributed storage system and is suitable for a distributed file system. 2) Structured data: Structured data refers to data that has a clear association relationship between them, and can usually be represented by a two-dimensional table, such as tabular data in a database. This type of data is stored in a relational database in a distributed storage system and supports SQL queries and transaction operations. 3) Semi-structured data: Semi-structured data is between unstructured and structured data, and has a certain structure but is not as strict as structured data. This type of data may contain some tags or metadata to describe the structure of the data, but it is still relatively flexible overall.

[0076] In one embodiment, the node load information includes multiple load information of the corresponding write node, and the comprehensive load score of the corresponding write node is calculated according to the node load information, including:

[0077] Obtaining load assessment criteria, wherein the load assessment criteria include multiple load information scoring criteria;

[0078] Matching a plurality of load information with corresponding load information scoring criteria to obtain a plurality of load information scores;

[0079] The comprehensive load score of the corresponding write node is calculated based on the multiple load information scores.

[0080] In one embodiment, multiple load information is matched with corresponding load information scoring criteria to obtain multiple load information scores, including:

[0081] Matching the processor usage with the processor usage scoring standard to obtain a processor usage matching result, wherein the processor usage matching result includes a processor usage score;

[0082] Matching the memory usage with the memory usage scoring standard to obtain a memory usage matching result, wherein the memory usage matching result includes a memory usage score;

[0083] Matching the actual operation status with the actual operation status scoring standard to obtain an actual operation status matching result, wherein the actual operation status matching result includes an actual operation status score;

[0084] The business requirement satisfaction is matched with the business requirement satisfaction scoring standard to obtain a business requirement satisfaction matching result, wherein the business requirement satisfaction matching result includes a business requirement satisfaction score.

[0085] Specifically, the load information is digitized through the above-mentioned scoring criteria so that the comprehensive load score can be calculated later, thereby being able to more comprehensively evaluate the load status of the system.

[0086] In a specific embodiment, the processor usage scoring standard is: 0%-20% is 1.0 point (low load, good performance); 21%-50% is 0.8 point; 51%-80% is 0.5 point (high load, need attention); 81%-100% is 0.2 point (extremely high load, may cause performance problems).

[0087] Memory usage scoring standard: 0%-30% is 1.0 point (low usage, sufficient resources); 31%-60% is 0.8 point (medium usage, normal); 61%-90% is 0.5 point (high usage, need to be monitored); 91%-100% is 0.2 point (extremely high usage, may cause memory overflow).

[0088] Scoring criteria for actual operation (based on log analysis, error rate, response time, etc.): No error logs and fast response time is 1.0 point; a small number of error logs and acceptable response time is 0.8 point; a large number of error logs and occasional extended response time is 0.5 point; frequent errors and significantly extended response time is 0.2 point.

[0089] Scoring criteria for business demand satisfaction (based on business indicators such as throughput, number of concurrent users, service quality, etc.): If business indicators are fully met and user satisfaction is high, it is 1.0 point; if business indicators are basically met with occasional small fluctuations, it is 0.8 point; if business indicators are partially not met and user feedback is delayed or unstable, it is 0.5 point; if business indicators are seriously not met and user experience is poor, it is 0.2 point.

[0090] In one embodiment, the comprehensive load calculation rule includes a processor usage weight, a memory occupancy weight, an actual operation weight, and a business demand satisfaction weight. The comprehensive load score of the corresponding write node is calculated according to the multiple load information scores, including:

[0091] Multiplying the processor usage score by the processor usage weight to obtain a first load score;

[0092] Multiply the memory usage score by the memory usage weight to obtain a second load score;

[0093] Multiply the actual operation condition score by the actual operation condition weight to obtain a third load score;

[0094] Multiply the business requirement satisfaction score by the business requirement satisfaction weight to obtain a fourth load score;

[0095] Add the first load score, the second load score, the third load score, and the fourth load score to obtain the comprehensive load score corresponding to the write node.

[0096] Specifically, obtain the comprehensive load score of each write node through weighted calculation to provide data support for load balancing, thereby ensuring the stable and efficient operation of the system.

[0097] In a specific embodiment, the comprehensive load score = (processor utilization rate score × 35% + memory occupancy rate score × 35% + actual operating condition score × 15% + business requirement satisfaction score × 15%) / total weight (100%). After obtaining the comprehensive load score, the weights can also be dynamically adjusted: flexibly adjust the weight distribution of each load information according to the actual application scenario and importance; threshold setting: fine-tune the above scoring criteria according to the specific system characteristics and business requirements; periodic evaluation: regularly conduct load evaluations to capture the changing trend of system performance over time; early warning mechanism: set a threshold for the comprehensive load score, and trigger an early warning when the score is lower than a preset value to take timely measures to optimize the system performance.

[0098] In one embodiment, obtain the data size corresponding to the target data type among multiple write nodes, and determine the target write node from the multiple write nodes according to the comprehensive load score and the data size corresponding to the target data type, including:

[0099] Obtain a first set threshold, and compare the comprehensive load scores of the multiple write nodes with the first set threshold to obtain a first comparison result;

[0100] Filter out multiple first write nodes according to the first comparison result, where the comprehensive load score of the first write node is less than or equal to the first set threshold;

[0101] Obtain a second set threshold, and compare the data size corresponding to the target data type among the multiple first write nodes with the second set threshold to obtain a second comparison result;

[0102] Filter out multiple second write nodes according to the second comparison result, where the data size corresponding to the target data type of the second write node is less than or equal to the second set threshold;

[0103] Sort the multiple second write nodes in descending order according to the data size corresponding to the target data type to obtain a data size sorting result;

[0104] According to the data size sorting result, take the second write node ranked first as the target write node.

[0105] Specifically, the first set threshold and the second set threshold can be average values, or arbitrary values ​​set according to actual conditions. First, multiple first write nodes whose comprehensive load scores are less than or equal to the first set threshold are screened out to improve system resource utilization; among the multiple first write nodes, multiple second write nodes whose data sizes corresponding to the target data type are less than or equal to the second set threshold are screened out to avoid the generation of small data files in the later stage, which causes additional performance overhead to the system. Among the multiple second write nodes, multiple second write nodes whose data sizes corresponding to the target data type are less than or equal to the second set threshold are screened out to ensure balanced data distribution while avoiding data dispersion as much as possible.

[0106] In one embodiment, the target data is written into the target write node, and the method further includes:

[0107] Determine whether the distributed storage system meets the load balancing condition, wherein the load balancing condition is that the difference in the comprehensive load scores of the distributed storage system is less than a third set threshold, and the difference in the comprehensive load scores is the difference between the maximum comprehensive load score and the minimum comprehensive load score in the distributed storage system;

[0108] In response to the distributed storage system not satisfying the load balancing condition, determining whether a node with a high comprehensive load score and a node with a low comprehensive load score store data of the same data type;

[0109] In response to the node with a high comprehensive load score and the node with a low comprehensive load score storing data of the same data type, transferring the data of the data type with the largest data size on the node with a high comprehensive load score to the node with a low comprehensive load score;

[0110] In response to the fact that the node with a high comprehensive load score and the node with a low comprehensive load score do not store data of the same data type, the smallest file on the node with a high comprehensive load score is transferred to the node with a low comprehensive load score.

[0111] Specifically, the high and low comprehensive load scores can be judged based on the same threshold, such as 0.5, or a high comprehensive load score means greater than 0.8, and a low comprehensive load score means less than 0.3. The above thresholds can be set specifically according to actual conditions. In response to the distributed storage system not meeting the load balancing condition, part of the data of the node with a high comprehensive load score is transferred to the node with a low comprehensive load score to solve the problem of excessive storage load on some nodes in the distributed storage system, and make the data distribution of each node more balanced as a whole.

[0112] In a specific embodiment, the load balancing conditions include: 1) Judgment based on node resource utilization. Unbalanced processor utilization: When the processor utilization of some nodes is higher than a certain threshold (such as 80% or higher) for a long time, and the processor utilization of other nodes is relatively low (such as less than 50%), it can be considered that there is a processor load imbalance. Unbalanced memory utilization: If the memory utilization of some nodes is too high, while other nodes have sufficient memory, this also indicates that there is a memory load imbalance. Unbalanced disk input and output utilization: When the disk input and output of some nodes are busy, while other nodes are relatively idle, it can be considered that there is a disk input and output load imbalance. 2) Judgment based on the actual data traffic and request processing capabilities of the distributed storage service.

[0113] In a specific embodiment, a method for calculating the load adjustment amount includes: calculating a comprehensive load average value based on the comprehensive load scores of multiple write nodes; subtracting the comprehensive load average value from the comprehensive load score of the node with unbalanced load to obtain a comprehensive load difference; and calculating the comprehensive load difference according to a set ratio to obtain the load adjustment amount.

[0114] In one embodiment, after writing the target data into the target write node, the method further includes:

[0115] Regularly perform health checks on multiple nodes to obtain health check results, where health checks include hardware status checks, software version checks, and network connectivity checks;

[0116] In response to the health check result indicating that a node fails and / or is abnormal, determine the failed node and / or abnormal node according to the node identification information in the health check result;

[0117] Obtain data from faulty nodes and / or abnormal nodes, and migrate the data from faulty nodes and / or abnormal nodes to other healthy nodes according to data migration rules;

[0118] Among them, data migration rules include data integrity verification and data compatibility verification. Data integrity verification includes digital signature verification, and data compatibility verification includes version compatibility verification.

[0119] Specifically, by performing regular health checks and fault recovery on each node, the high availability and data security of the system are ensured.

[0120] In a specific embodiment, the integrity, consistency and security of the data need to be ensured during the data migration process. Node failure generally refers to a situation where the node cannot work properly or fails completely, which may be caused by a variety of reasons, including but not limited to: Hardware failure: hard disk damage, physical damage or logical errors in the hard disk, resulting in the inability to read or write data. Memory failure: problems with the memory bar, resulting in system crashes or data loss. Power failure: insufficient or unstable power supply, resulting in the node being unable to start or unstable operation. Network interface card problem: the network interface card fails, affecting the node's network connection. Software error: operating system vulnerability, security vulnerabilities or errors in the operating system, resulting in service interruptions or data inconsistencies. Application crash: a serious error in the application causes the node to be unable to continue to provide services. Database connection problem: the database connection is interrupted or unstable, affecting data access and processing. Network problem: the network connection is interrupted, and the network connection between the node and other nodes or clients is disconnected. Network delay or packet loss: there is a delay or data loss during network transmission, affecting the normal transmission and access of data.

[0121] Performance anomalies usually mean that although the node can work normally, its performance does not meet expectations or standards. Possible manifestations include: slow response time, insufficient number of connections resulting in connection queuing, affecting the response time of requests. Slow SQL query, slow database query speed, resulting in data access delays. Slow external interface calls, slow interface calls with other systems or services. High resource utilization, high processor consumption, the processor occupancy rate is too high when the node processes requests, affecting the execution of other tasks. High memory usage, excessive node memory usage, which may lead to memory leaks or insufficient memory. High disk input and output: The node disk read and write operations are frequent, affecting the data access speed.

[0122] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0123] Embodiment 2

[0124] In one embodiment, Figure 3 As shown, a load balancing device is provided, the device comprising:

[0125] A first acquisition module, the first acquisition module is used to acquire multiple nodes in the distributed storage system in response to receiving a data write request, wherein the data write request includes target data and a target data type;

[0126] A first determination module, the first determination module is used to determine a plurality of write nodes from a plurality of nodes according to a target data type;

[0127] A collection module, the collection module is used to collect node load information of multiple write nodes;

[0128] A calculation module, which is used to calculate the comprehensive load score of the corresponding write node according to the node load information;

[0129] A second acquisition module, the second acquisition module is used to acquire the data size corresponding to the target data type in multiple write nodes;

[0130] A second determination module, the second determination module is used to determine a target write node from multiple write nodes according to the comprehensive load score and the data size corresponding to the target data type;

[0131] The write module is used to write target data to a target write node in response to a data write request.

[0132] In one embodiment, the first determining module is used to:

[0133] Obtaining data type distribution in a plurality of nodes, wherein the data type distribution represents a plurality of data type types included in the corresponding nodes;

[0134] Match the data type distribution of multiple nodes with the target data type to obtain a type matching result;

[0135] In response to the type matching result indicating that the corresponding node does not contain the data type kind of the target data type, determining the corresponding node as another node;

[0136] In response to the type matching result indicating that the corresponding node contains the data type kind of the target data type, the corresponding node is determined as a write node.

[0137] In one embodiment, the node load information includes multiple load information corresponding to the write node, and the calculation module is used to:

[0138] Obtaining load assessment criteria, wherein the load assessment criteria include multiple load information scoring criteria;

[0139] Matching a plurality of load information with corresponding load information scoring criteria to obtain a plurality of load information scores;

[0140] The comprehensive load score of the corresponding write node is calculated based on the multiple load information scores.

[0141] In one embodiment, the computing module is used to:

[0142] Matching the processor usage with the processor usage scoring standard to obtain a processor usage matching result, wherein the processor usage matching result includes a processor usage score;

[0143] Matching the memory usage with the memory usage scoring standard to obtain a memory usage matching result, wherein the memory usage matching result includes a memory usage score;

[0144] Matching the actual operation status with the actual operation status scoring standard to obtain an actual operation status matching result, wherein the actual operation status matching result includes an actual operation status score;

[0145] The business requirement satisfaction is matched with the business requirement satisfaction scoring standard to obtain a business requirement satisfaction matching result, wherein the business requirement satisfaction matching result includes a business requirement satisfaction score.

[0146] In one embodiment, the comprehensive load calculation rule includes a processor usage weight, a memory occupancy weight, an actual operation weight, and a business demand satisfaction weight, and the calculation module is used to:

[0147] Multiplying the processor usage score by the processor usage weight to obtain a first load score;

[0148] Multiply the memory usage score by the memory usage weight to obtain a second load score;

[0149] Multiply the actual operation condition score by the actual operation condition weight to obtain a third load score;

[0150] Multiply the business demand satisfaction score by the business demand satisfaction weight to obtain a fourth load score;

[0151] The first load score, the second load score, the third load score, and the fourth load score are added together to obtain a comprehensive load score of the corresponding write node.

[0152] In one embodiment, the second determining module is used to:

[0153] Obtaining a first set threshold, and comparing the comprehensive load scores of the plurality of write nodes with the first set threshold to obtain a first comparison result;

[0154] Screening out a plurality of first write nodes according to the first comparison result, wherein the comprehensive load score of the first write node is less than or equal to a first set threshold;

[0155] Obtaining a second set threshold, and comparing the data size corresponding to the target data type in the plurality of first write nodes with the second set threshold to obtain a second comparison result;

[0156] Filter out a plurality of second write nodes according to the second comparison result, wherein the data size corresponding to the target data type of the second write node is less than or equal to the second set threshold;

[0157] Sort the plurality of second write nodes from large to small according to the data sizes corresponding to the target data type, to obtain a data size sorting result;

[0158] According to the data size sorting result, the second write node ranked first is used as the target write node.

[0159] In one embodiment, after the writing module is used to write the target data into the target writing node, the device further includes:

[0160] A first judgment module, the first judgment module is used to judge whether the distributed storage system meets the load balancing condition, wherein the load balancing condition is that the difference of the comprehensive load scores of the distributed storage system is less than a third set threshold, and the difference of the comprehensive load scores is the difference between the maximum comprehensive load score and the minimum comprehensive load score in the distributed storage system;

[0161] A second judgment module, the second judgment module is used to judge whether the node with a high comprehensive load score and the node with a low comprehensive load score store data of the same data type in response to the distributed storage system not meeting the load balancing condition;

[0162] A first transfer module, the first transfer module is used for transferring data of the data type with the largest data size on the node with the high comprehensive load score to the node with the low comprehensive load score in response to the node with the high comprehensive load score and the node with the low comprehensive load score storing data of the same data type;

[0163] The second transfer module is used to transfer the smallest file on the node with a high comprehensive load score to the node with a low comprehensive load score in response to the node with a high comprehensive load score and the node with a low comprehensive load score not storing data of the same data type.

[0164] For the specific definition of the load balancing device, please refer to the definition of the load balancing method above, which will not be repeated here. Each module in the above load balancing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0165] Embodiment 3

[0166] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0167] In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type;

[0168] Determine a plurality of write nodes from the plurality of nodes according to the target data type;

[0169] Collect node load information of multiple write nodes, and calculate the comprehensive load score of the corresponding write node according to the node load information;

[0170] Obtaining data sizes corresponding to target data types from multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data sizes corresponding to the target data types;

[0171] Writes the target data to the target write node in response to the data write request.

[0172] When the program instructions are read and executed by one or more processors, they can also perform operations corresponding to the various steps in the above method embodiments. Please refer to the above description and will not be repeated here. Figure 4 , which exemplarily shows the architecture of the electronic device, which may specifically include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0173] Among them, the processor 410 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0174] The memory 420 can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage manager 424, and an icon font processing system 425, etc. can also be stored. The above-mentioned icon font processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0175] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0176] The network interface 414 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0177] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420).

[0178] In addition, the electronic device 400 can also obtain information on specific collection conditions from the virtual resource object collection condition information database 441 for use in condition judgment, etc.

[0179] It should be noted that, although the electronic device 400 only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the electronic device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0180] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling an electronic device (which can be a personal computer, a cloud server, or a network device, etc.) to execute the methods of various embodiments of the present application or certain parts of the embodiments.

[0181] Embodiment 4

[0182] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0183] In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type;

[0184] Determine a plurality of write nodes from the plurality of nodes according to the target data type;

[0185] Collect node load information of multiple write nodes, and calculate the comprehensive load score of the corresponding write node according to the node load information;

[0186] Obtaining data sizes corresponding to target data types from multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data sizes corresponding to the target data types;

[0187] Writes the target data to the target write node in response to the data write request.

[0188] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0189] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0191] Embodiment 5

[0192] In one embodiment, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0193] In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type;

[0194] Determine a plurality of write nodes from the plurality of nodes according to the target data type;

[0195] Collect node load information of multiple write nodes, and calculate the comprehensive load score of the corresponding write node according to the node load information;

[0196] Obtaining data sizes corresponding to target data types from multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data sizes corresponding to the target data types;

[0197] Writes the target data to the target write node in response to the data write request.

[0198] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer program product. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0199] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A load balancing method, applied to a distributed storage system, wherein the distributed storage system includes a plurality of nodes, characterized in that: The method comprises: In response to receiving a data write request, acquiring a plurality of nodes in the distributed storage system, wherein the data write request includes target data and a target data type; Determine a plurality of write nodes from a plurality of nodes according to the target data type; Collecting node load information of the multiple write nodes, and calculating a comprehensive load score of the corresponding write node according to the node load information; Obtaining data sizes corresponding to the target data type from the multiple write nodes, and determining a target write node from the multiple write nodes according to the comprehensive load score and the data size corresponding to the target data type; The target data is written to a target write node in response to the data write request.

2. The method according to claim 1, characterized in that: Determining a plurality of write nodes from a plurality of nodes according to the target data type includes: Obtaining data type distribution in the plurality of nodes, wherein the data type distribution represents a plurality of data type types included in the corresponding node; Matching the data type distribution of the multiple nodes with the target data type to obtain a type matching result; In response to the type matching result indicating that the corresponding node does not contain the data type type of the target data type, determining the corresponding node as another node; In response to the type matching result indicating that the corresponding node contains a data type type of the target data type, the corresponding node is determined as a write node.

3. The method according to claim 1, characterized in that: The node load information includes various load information of the corresponding write node, and the comprehensive load score of the corresponding write node is calculated according to the node load information, including: Obtaining a load evaluation standard, wherein the load evaluation standard includes a plurality of load information scoring standards; Matching the multiple load information with corresponding load information scoring criteria to obtain multiple load information scores; A comprehensive load score of the corresponding write node is calculated according to the multiple load information scores.

4. The method according to claim 3, characterized in that: The multiple load information are matched with corresponding load information scoring standards to obtain multiple load information scores, including: Matching the processor usage with the processor usage scoring standard to obtain a processor usage matching result, wherein the processor usage matching result includes a processor usage score; Matching the memory occupancy rate with the memory occupancy rate scoring standard to obtain a memory occupancy rate matching result, wherein the memory occupancy rate matching result includes a memory occupancy rate score; Matching the actual operation status with the actual operation status scoring standard to obtain an actual operation status matching result, wherein the actual operation status matching result includes an actual operation status score; The business requirement satisfaction is matched with the business requirement satisfaction scoring standard to obtain a business requirement satisfaction matching result, wherein the business requirement satisfaction matching result includes a business requirement satisfaction score.

5. The method according to claim 4, characterized in that: The comprehensive load calculation rule includes a processor usage weight, a memory occupancy weight, an actual operation weight, and a business demand satisfaction weight. The comprehensive load score of the corresponding write node is calculated according to the multiple load information scores, including: Multiplying the processor usage score by the processor usage weight to obtain a first load score; Multiplying the memory occupancy score by the memory occupancy weight to obtain a second load score; Multiplying the actual operation condition score by the actual operation condition weight to obtain a third load score; Multiplying the business demand satisfaction score by the business demand satisfaction weight to obtain a fourth load score; The first load score, the second load score, the third load score and the fourth load score are added together to obtain a comprehensive load score of the corresponding write node.

6. The method according to claim 1 or 5, characterized in that: Acquiring the data size corresponding to the target data type from the multiple write nodes, and determining the target write node from the multiple write nodes according to the comprehensive load score and the data size corresponding to the target data type, including: Obtaining a first set threshold, and comparing the comprehensive load scores of the plurality of write nodes with the first set threshold to obtain a first comparison result; Screening out a plurality of first write nodes according to the first comparison result, wherein the comprehensive load score of the first write nodes is less than or equal to a first set threshold; Obtaining a second set threshold, and comparing the data size corresponding to the target data type in the plurality of first write nodes with the second set threshold to obtain a second comparison result; Filter out a plurality of second write nodes according to the second comparison result, wherein the data size corresponding to the target data type of the second write node is less than or equal to a second set threshold; Sort the plurality of second write nodes from large to small according to the data sizes corresponding to the target data type, to obtain a data size sorting result; According to the data size sorting result, the second write node ranked first is used as the target write node.

7. The method according to claim 1, characterized in that: Writing the target data into a target write node, the method further comprises: Determine whether the distributed storage system meets the load balancing condition, wherein the load balancing condition is that the difference in the comprehensive load scores of the distributed storage system is less than a third set threshold, and the difference in the comprehensive load scores is the difference between the maximum comprehensive load score and the minimum comprehensive load score in the distributed storage system; In response to the distributed storage system not satisfying the load balancing condition, determining whether a node with a high comprehensive load score and a node with a low comprehensive load score store data of the same data type; In response to the node with a high comprehensive load score and the node with a low comprehensive load score storing data of the same data type, transferring the data of the data type with the largest data size on the node with a high comprehensive load score to the node with a low comprehensive load score; In response to the node with the high comprehensive load score and the node with the low comprehensive load score not storing data of the same data type, the smallest file on the node with the high comprehensive load score is transferred to the node with the low comprehensive load score.

8. A load balancing device, characterized in that: The device comprises: A first acquisition module, the first acquisition module is used to acquire multiple nodes in the distributed storage system in response to receiving a data write request, wherein the data write request includes target data and a target data type; A first determining module, the first determining module is used to determine a plurality of write nodes from a plurality of nodes according to the target data type; A collection module, the collection module is used to collect node load information of the multiple write nodes; A calculation module, the calculation module is used to calculate the comprehensive load score of the corresponding write node according to the node load information; A second acquisition module, the second acquisition module is used to acquire the data size corresponding to the target data type in the multiple write nodes; A second determination module, the second determination module is used to determine a target write node from a plurality of write nodes according to the comprehensive load score and a data size corresponding to the target data type; A write module is used to write the target data into a target write node in response to the data write request.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images