Distributed storage data self-adaptive load balancing method and system

Abstract

The invention relates to the technical field of distributed storage, and particularly discloses a distributed storage data self-adaptive load balancing method and system, and the method comprises the steps: distributing data to each node in distributed storage based on a consistent Hash algorithm; key index values of all the nodes are collected in real time, whether the key index values of all the nodes exceed a first set threshold value or not is judged, or whether the response time efficiency of key data obtaining exceeds a second set threshold value or not is judged; if yes, evaluating and sorting the performance of each node in the distributed storage based on an entropy weight method and TOPSIS to obtain a sorting result; positioning nodes with better performance based on a sorting result; migrating the data or the key data under the nodes of which the key index values exceed a first set threshold value to the nodes with better performance; according to the method, intelligent migration of the data to the low-load node is realized, so that the problem of non-uniform data distribution is effectively improved, and the overall performance and reliability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage systems, and in particular to a method and system for adaptive load balancing of distributed storage data. Background Art

[0002] With the rapid development of information technology, the storage and management of big data has become an important cornerstone of enterprise development. With the explosive growth of data volume, traditional centralized storage methods have gradually exposed performance bottlenecks and scalability issues. Distributed storage systems have become the preferred solution for processing large-scale data sets due to their high availability, scalability, and fault tolerance. However, in a distributed environment, how to effectively achieve data load balancing and ensure efficient use of system resources is a major challenge in current research and application.

[0003] Currently, load balancing strategies in distributed storage systems mostly rely on simple polling, random or fixed-weight-based allocation methods. These methods are unable to cope with complex and changeable load conditions and cannot accurately reflect the actual performance status of the nodes, resulting in uneven data distribution, excessive load on some nodes and idle resources on other nodes, affecting the overall performance and response time of the system, which not only affects the user experience, but may also have an impact on the operation of the enterprise. Summary of the invention

[0004] In view of the above problems, an object of the present invention is to provide a method for adaptive load balancing of distributed storage data, which comprehensively and accurately evaluates storage nodes by comprehensively considering performance indicators in multiple dimensions; at the same time, combined with real-time monitoring of node performance data, it realizes intelligent migration of data to low-load nodes, thereby effectively improving the problem of uneven data distribution and improving the overall performance and reliability of the system.

[0005] The second object of the present invention is to provide a system for adaptive load balancing of distributed storage data.

[0006] The first technical solution adopted by the present invention is: a method for adaptive load balancing of distributed storage data, comprising the following steps: S100: Distribute data to each node in the distributed storage system based on the consistent hashing algorithm; S200: collecting the key indicator value of each node in real time, judging whether the key indicator value of each node exceeds the first set threshold value, and if the key indicator value of any node exceeds the first set threshold value, executing step S300; or collecting the response time of obtaining key data, judging whether the response time of obtaining key data exceeds the second set threshold value, and if it exceeds the second set threshold value, executing step S300; S300: Evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; S400: Locate a node with better performance based on the sorting result; and migrate the data under the node whose key indicator value exceeds a first set threshold to the node with better performance, or migrate the key data under the node whose response time for obtaining key data exceeds a second set threshold to the node with better performance.

[0007] Preferably, the step S100 includes the following sub-steps: S110: Calculate the hash value of each node in the distributed storage and the hash value of each data using a hash function; S120: Mapping each node to a virtual node, distributing each virtual node on a virtual hash ring based on a hash value of each node; and mapping each data to each virtual node based on each data and the hash value of each node; S130: Storing the data mapped on each virtual node to the distributed node corresponding to each virtual node.

[0008] Preferably, the step S300 includes the following sub-steps: S310: Determine the evaluation index of each node, collect the node index value; and pre-process the node index value; S320: Based on the preprocessed node index value, the entropy value of each evaluation index is calculated using the entropy weight method; S330: performing difference coefficient calculation and normalization calculation on the entropy values ​​of the evaluation indicators to obtain the weight of each evaluation indicator; S340: Based on the weights of the evaluation indicators and the node indicator values, TOPSIS is used to comprehensively evaluate the performance of each distributed node to obtain the relative proximity of each distributed node; and all distributed nodes are sorted based on the relative proximity of each distributed node to obtain a sorting result.

[0009] Preferably, the step S310 further includes: saving the node indicator value in a time series database.

[0010] Preferably, the evaluation index of each node includes the hardware resource index, processing capacity index, network performance index and operation stability index of the node; The node indicator values ​​include one or more of the number of cores, number of threads, main frequency, CPU usage, total memory capacity, memory usage, storage type, total storage capacity, throughput, storage usage, data read / write speed, request response time, network interface type, network bandwidth, network delay, packet loss rate, network throughput, system health status, critical error information, number of restarts, node normal operation time and node failure recovery time.

[0011] Preferably, preprocessing the node index value includes: performing data cleaning, index positive transformation, and data dimensionless processing operations on the node index value.

[0012] Preferably, the entropy value of each evaluation index is calculated by the following formula: ; In the formula, e j is the entropy value of the evaluation index; p ij is the weight of the jth indicator of the i-th node; n is the number of nodes; m is the number of evaluation indicators; i is all values ​​from 1 to n; j is all values ​​from 1 to m.

[0013] Preferably, the weight of each evaluation index is calculated by the following formula: ; In the formula, w j is the weight coefficient of the j-th evaluation index; g j is the coefficient of difference of the j-th evaluation index; g k is the difference coefficient of each indicator of each evaluation object; m is the number of evaluation indicators; k is all values ​​from 1 to m.

[0014] Preferably, the relative proximity of each distributed node is expressed by the following formula: ; In the formula, f i is the relative proximity value of the distributed nodes; is the specific distance of the distributed node relative to the positive ideal solution; is the specific distance of the distributed node relative to the negative ideal solution.

[0015] The second technical solution adopted by the present invention is: a distributed storage data adaptive load balancing system, including a data distribution module, a monitoring module, an evaluation module and a data migration module; The data distribution module is used to distribute data to each node in the distributed storage system based on a consistent hashing algorithm; The monitoring module is used to collect the key indicator value of each node in real time, determine whether the key indicator value of each node exceeds a first set threshold, and if the key indicator value of any node exceeds the first set threshold, call the evaluation module; or collect the response time of obtaining key data, determine whether the response time of obtaining key data exceeds a second set threshold, and if it exceeds the second set threshold, call the evaluation module; The evaluation module is used to evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; The data migration module is used to locate nodes with better performance based on the sorting results; and to migrate data under nodes whose key indicator values ​​exceed a first set threshold to nodes with better performance, or to migrate key data under nodes whose response time for obtaining key data exceeds a second set threshold to nodes with better performance.

[0016] Beneficial effects of the above technical solution: (1) The method of adaptive load balancing of distributed storage data provided by the present invention aims to optimize the data distribution between storage nodes through intelligent algorithms and strategies, thereby improving the overall performance and stability of the system. This method is widely applicable to scenarios that require efficient data processing, and is of great significance for improving enterprise data processing capabilities and reducing operating costs.

[0017] (2) The present invention proposes a distributed storage data adaptive load balancing method based on entropy weight method and TOPSIS. This method comprehensively and accurately evaluates storage nodes by comprehensively considering performance indicators in multiple dimensions. At the same time, combined with real-time monitoring of node performance data, it realizes intelligent migration of data to low-load nodes, thereby effectively improving the problem of uneven data distribution and improving the overall performance and reliability of the system.

[0018] (3) The present invention improves system performance: through adaptive load balancing, hot spot data is effectively distributed to low-load nodes, thereby reducing single-point overload and improving data access speed and system response time.

[0019] (4) The present invention enhances system stability: the load balancing strategy reduces the probability of node failure and improves the stability and reliability of the system.

[0020] (5) The present invention optimizes resource utilization: The present invention can dynamically adjust data distribution according to the actual performance and load conditions of the nodes, so that system resources can be used more reasonably.

[0021] (6) The present invention simplifies management and maintenance: Adaptive load balancing reduces the need for manual intervention and simplifies system management and maintenance work.

[0022] (7) The distributed storage data adaptive load balancing method of the present invention is an efficient, intelligent and reliable solution, which is of great significance for improving the performance and stability of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic flow chart of a method for adaptive load balancing of distributed storage data provided by an embodiment of the present invention; Figure 2 A schematic diagram of consistent hashing data distribution provided for one embodiment of the present invention; Figure 3 A schematic diagram of performance evaluation and ranking of each node provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a system for adaptive load balancing of distributed storage data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0025] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0026] Embodiment 1 like Figure 1 As shown, an embodiment of the present invention provides a method for adaptive load balancing of distributed storage data, comprising the following steps: S100: Distribute data to each node in the distributed storage system based on the consistent hashing algorithm; like Figure 2 As shown, in a distributed storage system, the present invention uses a consistent hashing algorithm as a data distribution and load balancing technology to distribute data to each node in the distributed storage system, including the following sub-steps: S110: Calculate the hash value of each node in the distributed storage system and the hash value of each data by using a hash function; S120: Mapping each node to a virtual node, distributing each virtual node on a virtual hash ring based on a hash value of each node; and mapping each data to each virtual node based on each data and the hash value of each node; Mark these hash values ​​on the virtual hash ring. The value range of the hash space is 0~2 32 -1; based on the hash value of the data, the data whose hash value is close to the hash value of the node is mapped to the first corresponding virtual node in the clockwise direction of the hash ring; for example, the hash values ​​of the nodes are 3, 4, 5, 7, 10, etc., and the data with a hash value of 6 is mapped to the virtual node with a hash value of 7; this mapping method ensures that the nodes and data are basically evenly distributed on the hash ring.

[0027] The hash value of a node determines its position on the ring, while the hash value of a data item is used to determine its position on the ring and is stored in the first corresponding node clockwise on the hash ring; when a new node joins the system, the node calculates its hash value and locates it on the hash ring.

[0028] The data item will be migrated to the part that the new node is responsible for, usually to the new node closest to it; conversely, when a node leaves the system, the hash value of the node will be deleted from the hash ring, and the data originally responsible for the node will be transferred to the next node in the clockwise direction.

[0029] S130: Storing the data mapped on each virtual node to the distributed node corresponding to each virtual node; The data mapped on each virtual node is stored in the actual distributed node corresponding to each virtual node.

[0030] S200: collecting the key indicator value of each node in real time, judging whether the key indicator value of each node exceeds the first set threshold value, if the key indicator value of any node exceeds the first set threshold value, executing step S300; or judging whether the response time of obtaining key data exceeds the second set threshold value, if it exceeds the second set threshold value, executing step S300; The present invention automatically performs migration mechanism evaluation, including: real-time collection of key indicator values ​​of each node, and judging whether the key indicator value of each node exceeds a first set threshold value. If the key indicator value of any node exceeds the first set threshold value, it means that the node is overloaded and the performance of the node is poor, and the data of the node needs to be migrated to improve the performance of the node; the key indicator values ​​of the node include but are not limited to CPU, memory, disk capacity, response time, etc.

[0031] Or determine whether the response time for obtaining key data exceeds a second set threshold. If it exceeds the second set threshold, it means that the node storing the key data is overloaded and the performance of the node is poor. The key data of the node needs to be migrated to improve the response time of the key data.

[0032] S300: Evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; like Figure 3 As shown in the figure, the entropy weight method and TOPSIS are used to evaluate and rank the performance of each node in the distributed storage system, including the following sub-steps: S310: Determine the evaluation index of each node, collect the node index value; and pre-process the node index value; (1) Determine the evaluation index of each node; Before conducting a comprehensive evaluation, we must first clarify the object to be evaluated and determine the evaluation indicators related to the object; the evaluation indicators should be objective, quantifiable and representative, and there should be a certain degree of independence between the indicators; the object of this evaluation is each storage node in the distributed storage.

[0033] The evaluation indicators of each node include the hardware resource indicators, processing capacity indicators, network performance indicators, operation stability indicators, etc. of the node; among them, the hardware resource indicators include but are not limited to CPU indicators, memory indicators, storage indicators, CPU indicators include, for example, the number of cores, the number of threads, the main frequency, CPU utilization, etc., memory indicators include, for example, the total memory capacity, memory utilization, etc.; storage indicators include, for example, storage type, total storage capacity, throughput, storage utilization, etc.; Processing capacity indicators include but are not limited to data read / write speed and request response time; Network performance indicators include but are not limited to network interface type, network bandwidth, network latency, packet loss rate, and network throughput; Operation stability indicators include but are not limited to system health status, critical error information, number of restarts, node uptime, and node failure recovery time.

[0034] Hardware resources and processing power reflect the resource abundance of the node, network bandwidth and response time (i.e., network performance indicators) are directly related to data transmission efficiency and access speed, and operation stability indicators ensure the stability and availability of data during node operation. By collecting data on these performance indicators in real time and weighting them to find the degree of proximity to the ideal solution, a score ranking that reflects the comprehensive performance of the node is obtained.

[0035] (2) Collect node index values; In order to ensure the accuracy and timeliness of the data, the present invention adopts efficient data collection tools to collect the index values ​​of distributed storage nodes in real time, obtain the most timely and accurate index information, and dynamically monitor key performance indicators to conduct a comprehensive performance evaluation.

[0036] The collected node indicator values ​​include but are not limited to the number of cores, number of threads, main frequency, CPU usage, total memory capacity, memory usage, storage type, total storage capacity, throughput, storage usage, data read / write speed, request response time, network interface type, network bandwidth, network latency, packet loss rate, network throughput, system health status, critical error information, number of restarts, node normal operation time, node failure recovery time, etc.

[0037] Data collection can not only monitor the status indicators of the node itself, but also flexibly adjust the frequency and granularity of data collection, and optimize it according to actual conditions to avoid the burden on system performance caused by too frequent collection.

[0038] Furthermore, in one embodiment, it also includes: saving the node indicator value in a time series database; as a database specifically used to process time series data, the time series database has efficient data insertion and query capabilities, and is suitable for storing and managing collected performance indicators; by storing data in a time series database, the present invention can achieve rapid backtracking and analysis of historical data, thereby facilitating trend analysis and problem troubleshooting.

[0039] (3) Preprocessing the node index value; The preprocessing of node index values ​​includes: data cleaning, index positive transformation, data dimensionless processing and other operations on the node index values; data cleaning removes outliers and missing values ​​to ensure the accuracy of the data; data dimensionless transformation converts data of different dimensions into the same scale to avoid deviations in the calculation of index weights.

[0040] 1) Data cleaning; Data cleaning is the first step of data preprocessing, which aims to remove outliers and missing values ​​in the data to ensure the accuracy and reliability of the data. During the node performance evaluation process, data collection may be affected by various factors, such as network delays, system failures, or human errors, which may cause outliers in the collected data.

[0041] Outlier detection: Outliers usually refer to points in a data set that deviate significantly from other observations. By setting a reasonable value range, data points that do not conform to the expected pattern are identified and deleted or replaced to avoid affecting the overall performance evaluation.

[0042] Missing value processing: Missing values ​​are a common problem in data collection. Data may not be recorded due to various reasons. Processing is performed based on the importance of the indicators to the evaluation results, including deleting records with missing values, filling with the mean, etc.

[0043] 2) Evaluation indicators are becoming more positive; In data analysis and performance monitoring, many evaluation indicators (such as CPU usage, memory usage, network latency, etc.) are better as small as possible; in order to facilitate data application, the present invention performs positive processing on these evaluation indicators; positive evaluation indicator conversion converts the original evaluation indicators so that all evaluation indicators are expressed in the form of "the larger the better". This process not only helps to improve interpretability, but also optimizes the subsequent analysis results. The positive evaluation indicator is expressed by the following formula: ; In the formula, is the new evaluation index value after positive transformation; is the original value of the evaluation index; m is the number of evaluation indicators; Indicates the maximum value of the evaluation index; through the difference operation between the maximum value and the original value, the new evaluation index after positive conversion is obtained, and the smaller the better evaluation index is converted to unify all evaluation indicators.

[0044] 3) Dimensionless data processing; Data dimensionlessness is the process of converting data of different dimensions into the same scale to avoid deviation in the calculation of evaluation index weights; dimensionlessness allows different features to be compared within the same range, thereby eliminating errors caused by different dimensions; such processing can maintain the distribution of the original data while making different features comparable.

[0045] The dimensionless data processing uses square sum normalization, which is expressed by the following formula: ; In the formula, p ij is the proportion of the jth index of the i-th node; x ij is the original value of the jth index of the i-th node; a jk is the index value of the jth evaluation index of the kth node; n is the number of nodes; m is the number of evaluation indicators. By calculating the proportional relationship between the evaluation index value of each node and the sum of all node values ​​under the evaluation index, the relative importance of the node evaluation index in the whole is reflected.

[0046] S320: Based on the preprocessed node index value, an entropy weight method is used to calculate the entropy value of each evaluation index, so as to evaluate the importance of each evaluation index; In the entropy weight method, the entropy value is used to measure the degree of dispersion and difference of the evaluation indicators; when conducting a comprehensive evaluation, it is necessary to conduct a quantitative analysis of each evaluation indicator to ensure the scientificity and accuracy of the evaluation results. According to the definition of information entropy, the degree of dispersion of each evaluation indicator in the evaluation system can be measured by calculating the entropy value of each evaluation indicator. The entropy value of the evaluation indicator is calculated by the following formula: ; In the formula, e j is the entropy value of the evaluation index; p ij is the weight of the jth indicator of the i-th node; n is the number of nodes; m is the number of evaluation indicators; i is all values ​​from 1 to n; j is all values ​​from 1 to m.

[0047] The calculation of entropy value can help understand the relative importance of each evaluation indicator in the decision-making process; when the entropy value of an evaluation indicator is high, it means that the evaluation indicator provides rich information and its changes have a greater impact on the decision-making results; conversely, when the entropy value is low, it means that the evaluation indicator provides less information and has less impact on the decision.

[0048] S330: performing difference coefficient calculation and normalization calculation on the entropy value of each evaluation index to obtain the weight of each evaluation index; The weight of each evaluation indicator (i.e., indicator weight) is the contribution of the indicator to the comprehensive evaluation result. The greater the weight, the greater the impact of the evaluation indicator on the evaluation result. The steps for calculating the weight of each evaluation indicator include: Calculate the difference coefficient of the j-th evaluation indicator, determine the weight coefficient of the j-th evaluation indicator based on the difference coefficient of the j-th evaluation indicator, and then determine the importance of each evaluation indicator; the difference coefficient of the j-th evaluation indicator is expressed by the following formula: ; In the formula, g j is the coefficient of difference of the j-th evaluation index; e j is the entropy value of the evaluation index; m is the number of evaluation indicators.

[0049] The difference coefficients of the evaluation indicators are normalized and calculated to obtain the weights of each evaluation indicator. The weights of each evaluation indicator are expressed by the following formula: ; In the formula, w jis the weight coefficient of the j-th evaluation index; g j is the coefficient of difference of the j-th evaluation index; g k is the difference coefficient of each indicator of each evaluation object; m is the number of evaluation indicators; k is all values ​​from 1 to m.

[0050] S340: Based on the weight of each evaluation index and the node index value, TOPSIS is used to comprehensively evaluate the performance of each distributed node to obtain the relative proximity of each distributed node; and all distributed nodes are sorted based on the relative proximity of each distributed node to obtain a sorting result; According to the weight and index value of each evaluation index, the distance from each distributed node to the positive ideal solution and the negative ideal solution is calculated, and the distributed nodes are sorted according to their relative proximity to the ideal solution to obtain the final node evaluation order.

[0051] The distance from each distributed node to the positive ideal solution is calculated by the following formula: ; The distance from each distributed node to the negative ideal solution is calculated using the following formula: ; In the formula, is the specific distance of the distributed node relative to the positive ideal solution (i.e., the optimal solution); is the specific distance of the distributed node relative to the negative ideal solution (i.e., the worst solution); w j is the weight coefficient of the j-th evaluation index; b ij is the index value of the jth index of the i-th node; is the positive ideal solution of the jth index; is the negative ideal solution of the j-th indicator; n is the number of nodes; m is the number of evaluation indicators for each node; i is the i-th node; j is the j-th indicator item.

[0052] By calculating the distance between each distributed node and the two ideal solutions, the relative proximity of each distributed node to the two ideal solutions is obtained. The relative proximity reflects the performance of each distributed node in the overall indicator system. The relative proximity of each distributed node is expressed by the following formula: ; In the formula, f i is the relative proximity value of the distributed nodes; is the specific distance of the distributed node relative to the positive ideal solution (i.e., the optimal solution); is the specific distance of the distributed node relative to the negative ideal solution (i.e., the worst solution).

[0053] All distributed nodes are sorted according to the relative proximity values ​​of each distributed node, so as to determine the sorting results of all nodes (i.e. the final evaluation order); based on the sorting results, the best and worst performing distributed systems can be identified, providing a basis for further decision-making and optimization.

[0054] The above multi-dimensional assessment not only fully considers the mutual influence of various factors, but also ensures the rationality and scientificity of weight distribution.

[0055] S400: Locate nodes with better performance based on the sorting results; and migrate data under nodes whose key indicator values ​​exceed a first set threshold to nodes with better performance, or migrate key data under nodes whose response time exceeds a second set threshold to nodes with better performance.

[0056] The sorting results can be used to effectively evaluate the performance of each node in the distributed storage system, identify nodes with better performance and locate nodes with poorer performance; based on the sorting results, the distribution of data can be dynamically adjusted to optimize overall performance.

[0057] In the process of implementing data migration, we must first determine the source node, target node, and data scope that needs to be migrated. During the resource idle window, data will be migrated from nodes with poor performance to nodes with good performance. After the migration is completed, the system will update the metadata records to ensure the accuracy of the data records. In this way, the system can ensure the smooth progress of data migration and minimize the impact on users.

[0058] In the process of allocating data using the consistent hashing algorithm, due to algorithm reasons or uneven server resources, some nodes may be overloaded while other nodes are relatively idle. In order to further improve the load balancing effect, the present invention proposes a node weight scoring mechanism for the above-mentioned multi-factor comprehensive evaluation. Through adaptive load balancing, hot spot data and data of high-load nodes are effectively dispersed to low-load nodes, thereby reducing single-point overload and improving data access speed and system response time.

[0059] The adaptive data load balancing mechanism adopted in the present invention can effectively optimize data distribution and maintain resource balance through real-time monitoring, multi-dimensional evaluation and dynamic adjustment; this mechanism dynamically adjusts the storage location of data according to the actual load of the node to maximize the use of system resources and improve the performance and stability of the overall system; with the continuous development of technology, the adaptive data load balancing mechanism will also continue to evolve, bringing more efficient and intelligent data management solutions, so as to better meet the needs of future data storage.

[0060] Embodiment 2 like Figure 4 As shown, an embodiment of the present invention provides a distributed storage data adaptive load balancing system, including a data distribution module, a monitoring module, an evaluation module and a data migration module; The data distribution module is used to distribute data to each node in the distributed storage system based on a consistent hashing algorithm; The monitoring module is used to collect the key indicator value of each node in real time, determine whether the key indicator value of each node exceeds a first set threshold, and call the evaluation module if the key indicator value of any node exceeds the first set threshold; or collect the response time of obtaining key data, determine whether the response time of obtaining key data exceeds a second set threshold, and call the evaluation module if it exceeds the second set threshold; The evaluation module is used to evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; The data migration module is used to locate nodes with better performance based on the sorting results; and to migrate data under nodes whose key indicator values ​​exceed a first set threshold to nodes with better performance, or to migrate key data under nodes whose response time for obtaining key data exceeds a second set threshold to nodes with better performance.

[0061] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0062] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0063] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0064] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0065] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0066] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for adaptive load balancing of distributed storage data, characterized in that: The following steps are involved: S100: Distribute data to each node in the distributed storage system based on the consistent hashing algorithm; S200: collecting the key indicator value of each node in real time, judging whether the key indicator value of each node exceeds the first set threshold value, and if the key indicator value of any node exceeds the first set threshold value, executing step S300; or collecting the response time of obtaining key data, judging whether the response time of obtaining key data exceeds the second set threshold value, and if it exceeds the second set threshold value, executing step S300; S300: Evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; S400: Locating a node with better performance based on the sorting result; And migrate the data under the node whose key indicator value exceeds the first set threshold to the node with better performance, or migrate the key data under the node whose response time for obtaining key data exceeds the second set threshold to the node with better performance.

2. The method for adaptive load balancing of distributed storage data according to claim 1, characterized in that: The step S100 includes the following sub-steps: S110: Calculate the hash value of each node in the distributed storage system and the hash value of each data using a hash function; S120: Map each node to a virtual node, and distribute each virtual node on a virtual hash ring based on the hash value of each node; and mapping each data to each virtual node based on the hash value of each data and each node; S130: Storing the data mapped on each virtual node to the distributed node corresponding to each virtual node.

3. The method for adaptive load balancing of distributed storage data according to claim 1, characterized in that: The step S300 includes the following sub-steps: S310: Determine the evaluation index of each node, collect the node index value; and pre-process the node index value; S320: Based on the preprocessed node index value, the entropy value of each evaluation index is calculated using the entropy weight method; S330: performing difference coefficient calculation and normalization calculation on the entropy values ​​of the evaluation indicators to obtain the weight of each evaluation indicator; S340: Based on the weights of the evaluation indicators and the node indicator value, TOPSIS is used to comprehensively evaluate the performance of each distributed node to obtain the relative proximity of each distributed node; And all the distributed nodes are sorted based on the relative proximity of the distributed nodes to obtain a sorting result.

4. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The step S310 also includes: saving the node indicator value in a time series database.

5. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The evaluation index of each node includes the node's hardware resource index, processing capacity index, network performance index and operation stability index; The node indicator values ​​include one or more of the number of cores, number of threads, main frequency, CPU usage, total memory capacity, memory usage, storage type, total storage capacity, throughput, storage usage, data read / write speed, request response time, network interface type, network bandwidth, network delay, packet loss rate, network throughput, system health status, critical error information, number of restarts, node normal operation time and node failure recovery time.

6. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The preprocessing of the node index value includes: performing data cleaning, index positive transformation, and data dimensionless processing operations on the node index value.

7. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The entropy value of each evaluation index is calculated by the following formula: ； In the formula, e j is the entropy value of the evaluation index; p ij is the weight of the jth indicator of the i-th node; n is the number of nodes; m is the number of evaluation indicators; i is all values ​​from 1 to n; j is all values ​​from 1 to m.

8. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The weight of each evaluation index is calculated by the following formula: ； In the formula, w j is the weight coefficient of the j-th evaluation index; g j is the coefficient of difference of the j-th evaluation index; g k is the difference coefficient of each indicator of each evaluation object; m is the number of evaluation indicators; k is all values ​​from 1 to m.

9. The method for adaptive load balancing of distributed storage data according to claim 3, characterized in that: The relative proximity of each distributed node is expressed by the following formula: ； In the formula, f i is the relative proximity value of the distributed nodes; is the specific distance of the distributed node relative to the positive ideal solution; is the specific distance of the distributed node relative to the negative ideal solution.

10. A distributed storage data adaptive load balancing system, characterized in that: It includes data allocation module, monitoring module, evaluation module and data migration module; The data distribution module is used to distribute data to each node in the distributed storage system based on a consistent hashing algorithm; The monitoring module is used to collect the key indicator value of each node in real time, determine whether the key indicator value of each node exceeds a first set threshold, and if the key indicator value of any node exceeds the first set threshold, call the evaluation module; or collect the response time of obtaining key data, determine whether the response time of obtaining key data exceeds a second set threshold, and if it exceeds the second set threshold, call the evaluation module; The evaluation module is used to evaluate and sort the performance of each node in the distributed storage system based on the entropy weight method and TOPSIS to obtain a sorting result; The data migration module is used to locate a node with better performance based on the sorting result; And migrate the data under the node whose key indicator value exceeds the first set threshold to the node with better performance, or migrate the key data under the node whose response time for obtaining key data exceeds the second set threshold to the node with better performance.

Images