Data balancing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data balancing method and device, equipment and a storage medium, which can be applied to the technical field of computers, in the method, multi-round iterative data migration is performed in a server set until an iteration stop condition is met, and a data balancing result of the server set is obtained. Each round of iteration comprises the step of obtaining a target server set comprising an emigration server set and an immigration server set from the server sets. And obtaining the migration data volume of each emigration server based on the first total data volume of the target server set and the first data difference quantity between the reference data volume corresponding to each round of iteration of the server set. And migrating the data of the emigration servers to the immigration server set according to the migration data volume of each emigration server. According to the method, data is transferred from a server with a high capacity ratio to a server with a low capacity ratio in a multi-round iteration manner by utilizing a peak clipping and valley filling principle, so that the problem of unbalanced data storage is effectively solved, and the efficiency of data balance is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data balancing method, apparatus, device and storage medium. Background Art

[0002] With the continuous expansion of business and the requirement of data collection, each business system backs up data to Hadoop Distributed File System (HDFS). Although this data backup method can meet the needs of data scalability and long-term storage.

[0003] However, in actual applications, a large amount of data is continuously generated and stored in distributed systems. When the data growth rate and business needs change, some nodes in the distributed file system will carry more data, while other nodes may be relatively idle, which will cause data storage imbalance. Data imbalance will cause some servers to be overloaded, while other server resources are not fully utilized, which will affect the performance and response time of the entire system and also lead to a waste of storage resources. Summary of the invention

[0004] The embodiments of the present application provide a data balancing method, apparatus, device and storage medium to achieve local and global balance of data stored in a distributed file system and reduce waste of storage resources.

[0005] On the one hand, an embodiment of the present application provides a data balancing method, the method comprising:

[0006] Perform multiple rounds of iterative data migration between servers in the server set until an iteration stop condition is met to obtain a data balance result of the server set, wherein each round of iteration includes the following operations:

[0007] Obtaining a target server set including a migration-out server set and a migration-in server set from the server set;

[0008] Obtaining the migration data volume of each migrated server in the migrated server set based on a first total data volume of the target server set and a first data difference volume between the target server set and a reference data volume corresponding to the server set in this iteration;

[0009] According to the amount of migrated data of each of the migrated-out servers, the data of each of the migrated-out servers is migrated to the set of migrated-in servers, and the amount of data of each of the migrated-out servers is updated.

[0010] On the one hand, an embodiment of the present application provides a data balancing device, the device comprising:

[0011] A processing module is used to perform multiple rounds of iterative data migration between servers in a server set until an iteration stop condition is met to obtain a data balance result of the server set. The processing module includes an acquisition module, a calculation module, and a migration module for executing each round of iteration. During each round of iteration:

[0012] The acquisition module is used to obtain a target server set including a migration-out server set and a migration-in server set from the server set.

[0013] The calculation module is used to obtain the migration data volume of each migrated server in the migrated server set based on the first total data volume of the target server set and the first data difference between the reference data volume corresponding to the server set in this round of iteration.

[0014] The migration module is used to migrate the data of each outgoing server to the incoming server set according to the amount of migrated data of each outgoing server, and to update the amount of data of each outgoing server.

[0015] Optionally, the acquisition module is specifically used for:

[0016] Sorting the multiple servers in the server set in ascending order of data volume to obtain a sorting result;

[0017] Based on the ranking result, selecting the first M servers from the multiple servers to form the migration-in server set; and selecting the last L servers from the multiple servers to form the migration-out server set, wherein M and L are both greater than 1;

[0018] The target server set is the union of the outgoing server set and the incoming server set.

[0019] Optionally, the calculation module is further used for:

[0020] Obtaining the average data volume of multiple servers in the server set during this round of iteration;

[0021] The mean value of the data volume is multiplied by the number of target servers in the target server set to obtain a reference data volume corresponding to the server set in this round of iteration.

[0022] Optionally, the calculation module is specifically used for:

[0023] Based on the first data difference amount and a preset adjustment coefficient, obtaining the total amount of migrated data of the set of migrated servers;

[0024] Based on the total amount of migrated data and the data migration ratio of each migrated-out server, the amount of migrated data of each migrated-out server is obtained.

[0025] Optionally, the calculation module is further used for:

[0026] Obtaining a second data difference between the data volume of each migrated server and an average of the data volumes of multiple servers in the server set;

[0027] Sum the second data difference amounts corresponding to a plurality of migrated servers in the migrated server set to obtain a total data difference amount;

[0028] The ratio of the second data difference amount corresponding to each of the migrated-out servers to the total data difference amount is used as the data migration ratio of each of the migrated-out servers.

[0029] Optionally, the migration module is specifically used to:

[0030] According to the amount of migrated data of each of the migrated servers and the amount of data of multiple migrated servers in the migrated server set, selecting at least one migrated server from the multiple migrated servers;

[0031] The data of each of the outgoing servers is migrated to the at least one incoming server.

[0032] Optionally, the migration module is further used for:

[0033] The data volume of the at least one migration server is updated.

[0034] Optionally, the processing module is further used to:

[0035] Based on the updated data volumes of the multiple servers in the server set, obtaining a system performance indicator for this iteration;

[0036] When the system performance index of the current iteration reaches a preset threshold, the iteration is stopped and the data volumes of the multiple servers in the server set after the current iteration are updated are used as the data balancing result of the server set.

[0037] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data balancing method when executing the program.

[0038] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned data balancing method.

[0039] In the embodiment of the present application, the peak-shaving and valley-filling principle is utilized to iteratively migrate data from servers with a higher capacity share (i.e., outbound servers) to servers with a lower capacity share (i.e., inbound servers) in multiple rounds. This strategy accurately locates the servers that require data migration, effectively solves the problem of unbalanced data storage in the distributed file system, greatly reduces the impact of balancing operations, and effectively improves the efficiency of data balancing. This method changes the traditional data balancing method and provides an efficient solution for data management in large-scale distributed systems.

[0040] Secondly, the optimization of data distribution of the distributed file system in this application is not done in one step, but gradually achieves data balance through continuous multiple rounds of iterations. In each round of iteration, data is migrated to local servers to achieve local data optimization of the system; through multiple rounds of iterations, the global data balance of the system is finally achieved, thereby improving the performance and response time of the entire system. Combining local optimization with global optimization better handles the complexity of large-scale data sets. This solution not only improves the balance of data distribution, but also ensures the stability of data distribution during the long-term operation of the system.

[0041] In addition, the method is more intelligent and flexible, reduces manual involvement, improves system operation efficiency, and reduces waste of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0043] Figure 2 A flowchart of a data balancing method provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of a method for calculating the amount of migration data provided in an embodiment of the present application;

[0045] Figure 4 A flowchart of a data balancing method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a data balancing method and device provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The following is a brief introduction to the system architecture diagram applicable to the technical solution of the embodiment of the present application. It should be noted that the process introduced below is only used to illustrate the embodiment of the present application and is not a limitation.

[0050] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application, and the system architecture at least includes a data balancing device 101 and a distributed file system 102.

[0051] The distributed file system 102 is a distributed storage framework including multiple servers, providing a high-throughput data access function and capable of storing and accessing data across multiple servers.

[0052] The data balancing device 101 may be a terminal device, and the terminal device 101 may be a smart phone, a smart watch, a tablet computer, a laptop computer, etc., but is not limited thereto. The data balancing device 101 may also be a server inside the distributed file system 102 or a server independent of the distributed file system 102. The data balancing device 101 may also be a data balancing tool deployed in the distributed file system 102, and this application does not make specific limitations on this.

[0053] In practical applications, facing hardware failures, which are common problems in large-scale distributed systems, this application proposes a stable fault-tolerant mechanism. When faced with abnormal situations, this fault-tolerant mechanism can ensure that the data balancing device 101 is restored to a stable operating state, effectively ensuring the reliability of the data. This real-time dynamic fault-tolerant mechanism can better cope with various complex abnormal situations compared to traditional static fault-tolerant measures.

[0054] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited to these.

[0055] The method in the embodiment of the present application can be executed by the data balancing device 101. In the embodiment of the present application, the data balancing device 101 and the distributed file system 102, and the multiple servers in the distributed file system 102 can be directly or indirectly connected through one or more networks. The network can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course other possible networks, which are not limited in the embodiment of the present application.

[0056] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a process of a data balancing method, such as Figure 2 As shown, the process of the method is executed by a computer device, which may be Figure 1 The data balancing device 101 shown includes the following steps:

[0057] Step 200, performing multiple rounds of iterative data migration between servers in the server set until an iteration stop condition is met, and obtaining a data balance result of the server set, wherein each round of iteration includes the following steps 201 to 203:

[0058] Step 201: Obtain a target server set including a migration-out server set and a migration-in server set from a server set.

[0059] Specifically, the server set refers to the set of all or part of the servers included in the distributed file system. The server set to be migrated out refers to the set of servers with a high capacity (i.e., a high load) in the server set. The server set to be migrated out adopts S high The migration server set refers to the server set with a lower capacity (i.e., lower load) in the server set. The migration server set adopts S low The union of the outgoing server set and the incoming server set is the target server set, that is, S target =S high ∪S low , where S target Represents a collection of target servers.

[0060] In some embodiments, the outgoing server set and the incoming server set are obtained by:

[0061] Sort multiple servers in the server set in ascending order of data volume to obtain a sorting result; according to the data volume sorting result, select the first M servers from the multiple servers to form an incoming server set; select the last L servers from the multiple servers to form an outgoing server set, where M and L are both greater than 1.

[0062] Specifically, the data volume of the server can be: the utilization rate of the server (also known as capacity ratio, that is, the ratio of the size of data stored by the server to the total data capacity that the server can store); the data volume of the server can also be the size of data stored by the server. The data volume of the server is used to characterize the load status of the server. The larger the data volume of the server, the higher the load of the server. The values ​​of M and L can be set according to actual conditions, and the values ​​of M and L can be equal or unequal. It should be noted that when selecting the server set to migrate into and the server set to migrate out, they can also be sorted in order from large to small in terms of data volume. This application does not make any specific restrictions on this.

[0063] In some embodiments, the outgoing server set and the incoming server set are obtained in the following manner: the data volume of multiple servers in the server set is obtained, and servers with data volumes greater than a first threshold are selected from the multiple servers to form the outgoing server set; and servers with data volumes less than a second threshold are selected from the multiple servers to form the incoming server set.

[0064] The first threshold and the second threshold can be set according to actual conditions. The first threshold is greater than the second threshold. The settings of the first threshold and the second threshold can be different in each round of iteration.

[0065] In an embodiment of the present application, during each round of iteration, some servers with excessively high and low data volumes are selected as migration-out servers and migration-in servers for subsequent iterations. This strategy accurately locates the servers that require data migration, greatly reduces the impact range of data balancing operations, and effectively achieves the efficiency of data balancing.

[0066] Step 202: Based on the first total data volume of the target server set and the first data difference between the reference data volume corresponding to the server set in this iteration, obtain the migration data volume of each migrated server in the migrated server set.

[0067] Specifically, the reference data volume corresponding to the server set in this round of iteration is obtained in the following way:

[0068] The average data volume of multiple servers in the server set in this round of iteration is obtained; the average data volume is multiplied by the number of target servers in the target server set to obtain the reference data volume corresponding to the server set in this round of iteration. Then, based on the first total data volume of the target server set, the first data difference between the reference data volume corresponding to the server set in this round of iteration is obtained, as shown in the following formula (1):

[0069]

[0070] Wherein, ΔC represents the difference of the first data; S target represents the target server set; S represents the server set; Capacity(i) represents the amount of data stored by target server i in the target server set; represents the first total data volume of the target server set; m represents the number of servers in the server set; n represents the number of target servers in the target server set, that is, the total number of migrated-in servers and migrated-out servers; Capacity(j) represents the data volume stored by server j in the server set; Indicates the amount of reference data corresponding to the server set in this iteration.

[0071] In order to explain the above formula (1) more accurately and intuitively, an example is given below where m is 10 and n is 4. The server set included in the distributed file system includes 10 servers, and 4 target servers are selected from the 10 server sets to form a target server set.

[0072] The first total data volume of the target server set is: the total data volume of data stored in the four target servers in the target server set. Represents the average data volume of the 10 servers in the server set. Multiply the average data volume of the 10 servers by the number of target servers to obtain the reference data volume corresponding to the server set in this round of iteration, that is,

[0073] Will and By subtracting them, the first data difference can be obtained.

[0074] In some embodiments, the amount of migrated data for each server in the set of migrated servers is obtained by: Figure 3 As shown, the following steps are included:

[0075] Step 301: Based on the first data difference amount and a preset adjustment coefficient, the total amount of migrated data of the migration-out server set is obtained.

[0076] Specifically, the preset adjustment coefficient may be a set system bandwidth, or a ratio of a system stability parameter to a current system state parameter, or other preset coefficients, and this application does not make any specific limitation on this.

[0077] The first data difference amount is multiplied by the preset adjustment coefficient to obtain the total amount of migrated data from the server set, as shown in the following formula (2):

[0078] D move =K×ΔC………………(2)

[0079] Wherein, ΔC represents the first data difference; K is a preset adjustment coefficient.

[0080] Step 302: Determine the data migration ratio of each server to be migrated out.

[0081] Specifically, the data migration ratio of each migrated server is obtained in the following way:

[0082] The second data difference between the data volume of each migrated server and the average data volume of multiple servers in the server set is obtained. The second data difference corresponding to multiple migrated servers in the migrated server set is summed to obtain the total data difference. The ratio between the second data difference corresponding to each migrated server and the total data difference is used as the data migration ratio of each migrated server. The above process can be expressed by the following formula (3):

[0083]

[0084] Where m represents the number of servers in the server set; S high represents the set of servers to be migrated out, Capacity(i) represents the amount of data stored in the server i to be migrated out; S represents the set of servers, Capacity(j) represents the amount of data stored in the server j in the set of servers.

[0085] That is, in the set of migrated servers, if the data volume of a migrated server is significantly different from the average data volume of multiple servers in the server set, the amount of migrated data of the migrated server will also be larger.

[0086] Step 303: Based on the total amount of migrated data and the data migration ratio of each migrated-out server, the amount of migrated data of each migrated-out server is obtained.

[0087] Specifically, the total amount of migrated data of the set of migrated servers calculated in step 301 is multiplied by the data migration ratio of each migrated server in step 302 to obtain the amount of migrated data of each migrated server. The specific calculation formula is shown in the following formula (4):

[0088]

[0089] Where m represents the number of servers in the server set; S high represents the set of servers to be migrated out, Capacity(i) represents the amount of data stored in the server i to be migrated out; S represents the set of servers, Capacity(j) represents the amount of data stored in the server j in the server set; DM(i) represents the amount of data migrated out of the server i; D move Indicates the total amount of data migrated from the server collection.

[0090] In the embodiment of the present application, the peak-shaving and valley-filling principle is used to iterate the data from the server with a higher capacity share to the server with a lower capacity share in multiple rounds. This strategy accurately locates the server that needs to migrate data, greatly reduces the impact range of the balancing operation, and effectively improves the efficiency of data balancing. This method changes the traditional data balancing method and provides an efficient solution for data management in large-scale distributed systems. Secondly, compared with the traditional method of achieving global balance at one time, the present application gradually achieves data balance of the server cluster through continuous multiple rounds of iterations, reduces the impact on the big data cluster, reduces the pressure on the server cluster, and is more convenient to deal with various complex abnormal situations.

[0091] Step 203: according to the amount of migrated data of each outgoing server, migrate the data of each outgoing server to the set of incoming servers, and update the amount of data of each outgoing server.

[0092] In some embodiments, at least one incoming server is selected from multiple incoming servers according to the amount of migrated data of each outgoing server and the amount of data of multiple incoming servers in the incoming server set; and the data of each outgoing server is migrated to the at least one incoming server.

[0093] Specifically, in the process of migration according to the amount of migration data of the outgoing server and the amount of data of multiple incoming servers in the incoming server set, the outgoing server and the incoming server can be migrated one-to-one, or one-to-many, many-to-many, or many-to-one. This application does not limit this.

[0094] In some embodiments, after migrating the data of each out-migrating server to the at least one in-migrating server, the data volume of the at least one in-migrating server is updated.

[0095] Specifically, while updating the data volume of at least one incoming server, the data volume of each outgoing server is also updated. The method of updating the data volume of the outgoing server is as shown in the following formula (5):

[0096] New_Capacity(i)=Capacity(i)-DM(i)………………(5)

[0097] Among them, New_Capacity(i) represents the data volume of the data saved after the migration server i is updated; Capacity(i) represents the data volume of the data saved before the migration server i is updated; DM(i) represents the migration data volume of the migration server.

[0098] In some embodiments, after the data volume of at least one migrated server is updated, the system performance index of this iteration is obtained based on the data volume of multiple servers in the updated server set. When the system performance index of this iteration reaches a preset threshold, that is, the iteration stop condition is met, the iteration is stopped and the data volume of multiple servers in the server set after the update of this iteration is used as the data balance result of the server set.

[0099] Specifically, the system performance indicator may be the utilization rate of each of the multiple servers in the server set. When the utilization rate of each of the multiple servers reaches a preset threshold, it means that the iteration stop condition is met in this round, and the iteration is stopped and the data volume of the multiple servers in the server set after the iteration is updated in this round is used as the data balance result of the server set.

[0100] Of course, in addition to the server utilization, the system performance indicators may also include other indicators such as load balancing and system stability. When the monitored system performance indicators include multiple indicators, the multiple indicators (for example, server utilization, load balancing, system stability, etc.) can be combined to obtain the comprehensive performance indicators of this round of iteration. When the comprehensive performance indicators of this round of iteration meet the preset conditions, it means that the iteration stop conditions are met, then the iteration is stopped and the data volume of multiple servers in the server set updated in this round of iteration is used as the data balance result of the server set.

[0101] In the embodiment of the present application, the peak-cutting and valley-filling principle is used to iteratively migrate data from servers with a higher capacity share to servers with a lower capacity share in multiple rounds. This strategy accurately locates the server that needs data migration, greatly reduces the impact range of the balancing operation, and effectively improves the efficiency of data balancing. This method changes the traditional data balancing method and provides an efficient solution for data management in large-scale distributed systems. Since the present application realizes the global balance of data stored in the distributed file system, it effectively reduces the risk of system failure caused by uneven data distribution, avoids server failure due to excessive load, and even the serious consequence of the entire cluster crashing.

[0102] Secondly, the optimization of data distribution of the distributed file system in this application is not done in one step, but gradually achieves data balance through continuous multiple rounds of iterations. In each round of iteration, data is migrated to local servers to achieve local data optimization of the system; through multiple rounds of iterations, the global data balance of the system is finally achieved, thereby improving the performance and response time of the entire system. Combining local optimization with global optimization better handles the complexity of large-scale data sets. This solution not only improves the balance of data distribution, but also ensures the stability of data distribution during the long-term operation of the system.

[0103] In addition, the method is more intelligent and flexible, reduces manual involvement, improves system operation efficiency, and reduces waste of storage resources.

[0104] In order to better explain the embodiment of the present application, a data balancing method provided by the embodiment of the present application is introduced below in combination with the HDFS data balancing system. The process of the method can be executed by the data balancing tool deployed in HDFS, including the following steps: Figure 4 As shown:

[0105] Step 401, initialization.

[0106] Specifically, it includes setting the termination condition, the target server number n, and obtaining the server set S consisting of all servers in HDFS.

[0107] 402, main cycle stage. The main cycle stage specifically includes the following steps 4021-4026:

[0108] Step 4021, select a target server set.

[0109] Specifically, multiple servers in the server set are sorted in order of data volume from small to large to obtain a sorting result. According to the data volume sorting result, the servers ranked in the top n / 2 positions are selected from the multiple servers to form the migration server set S low ; Select the servers ranked in the last n / 2 positions from multiple servers to form the migration server set S high The target server set is the union of the outgoing server set and the incoming server set.

[0110] Step 4022, calculate the capacity difference and adjustment factor.

[0111] Specifically, the capacity difference (ie, the first data difference amount) is calculated by the above formula (1), and the adjustment coefficient is set.

[0112] Step 4023, calculate the amount of migrated data.

[0113] Specifically, the total amount of migrated data of the set of migrated servers is calculated according to the capacity difference and the adjustment coefficient through the above formula (2); the data migration ratio of each migrated server is determined through the above formula (3), and the amount of migrated data of each migrated server is determined through the above formula (4).

[0114] Step 4024, data migration operation.

[0115] Specifically, according to the amount of migrated data of each out-migrating server calculated in step 4023, at least one incoming server is selected from the multiple incoming servers, and the data of the out-migrating servers is migrated to the at least one incoming server.

[0116] Step 4025, monitoring system performance indicators.

[0117] Specifically, the performance indicator of the system may be the utilization rate of each of the multiple servers in the server set.

[0118] Step 4026, determine whether the system performance index reaches the termination condition, if so, end the iteration, otherwise return to execute step 4021.

[0119] In the embodiment of the present application, the peak-shaving and valley-filling principle is utilized to iteratively migrate the data of the distributed file system from servers with a higher capacity share to servers with a lower capacity share in multiple rounds. Through this method of local balance and dynamic update of balance targets, efficient global balance is achieved, which significantly improves the data balance of the Hadoop distributed file system, greatly reduces the difficulty and frequency of personnel maintenance, and also improves the resource utilization of the Hadoop cluster, thereby improving the overall performance and stability of the system.

[0120] Based on the same technical concept, the present application embodiment provides a structural diagram of a data balancing device, such as Figure 5 As shown, the data balancing device 500 includes:

[0121] The processing module 501 is used to perform multiple rounds of iterative data migration between servers in the server set until the iteration stop condition is met to obtain the data balance result of the server set. The processing module includes an acquisition module 502, a calculation module 503, and a migration module 504 for performing each round of iteration; during each round of iteration:

[0122] The acquisition module 502 is used to obtain a target server set including a migration-out server set and a migration-in server set from the server set;

[0123] The calculation module 503 is used to obtain the migration data volume of each migrated server in the migrated server set based on the first total data volume of the target server set and the first data difference between the reference data volume corresponding to the server set in this round of iteration;

[0124] The migration module 504 is used to migrate the data of each outgoing server to the set of incoming servers according to the amount of migrated data of each outgoing server, and update the amount of data of each outgoing server;

[0125] Optionally, the acquisition module 502 is specifically used for:

[0126] Sorting the multiple servers in the server set in ascending order of data volume to obtain a sorting result;

[0127] Based on the ranking result, selecting the first M servers from the multiple servers to form the migration-in server set; and selecting the last L servers from the multiple servers to form the migration-out server set, wherein M and L are both greater than 1;

[0128] The target server set is the union of the outgoing server set and the incoming server set.

[0129] Optionally, the calculation module 503 is further used for:

[0130] Obtaining the average data volume of multiple servers in the server set during this round of iteration;

[0131] The mean value of the data volume is multiplied by the number of target servers in the target server set to obtain a reference data volume corresponding to the server set in this round of iteration.

[0132] Optionally, the calculation module 503 is specifically used for:

[0133] Based on the first data difference amount and a preset adjustment coefficient, obtaining the total amount of migrated data of the set of migrated servers;

[0134] Based on the total amount of migrated data and the data migration ratio of each migrated-out server, the amount of migrated data of each migrated-out server is obtained.

[0135] Optionally, the calculation module 503 is further used for:

[0136] Obtaining a second data difference between the data volume of each migrated server and an average of the data volumes of multiple servers in the server set;

[0137] Sum the second data difference amounts corresponding to a plurality of migrated servers in the migrated server set to obtain a total data difference amount;

[0138] The ratio of the second data difference amount corresponding to each of the migrated-out servers to the total data difference amount is used as the data migration ratio of each of the migrated-out servers.

[0139] Optionally, the migration module 504 is specifically used for:

[0140] According to the amount of migrated data of each of the migrated servers and the amount of data of multiple migrated servers in the migrated server set, selecting at least one migrated server from the multiple migrated servers;

[0141] The data of each of the outgoing servers is migrated to the at least one incoming server.

[0142] Optionally, the migration module 504 is further configured to:

[0143] The data volume of the at least one migration server is updated.

[0144] Optionally, the processing module 501 is further configured to:

[0145] Based on the updated data volumes of the multiple servers in the server set, obtaining a system performance indicator for this iteration;

[0146] When the system performance index of the current iteration reaches a preset threshold, the iteration is stopped and the data volumes of the multiple servers in the server set after the current iteration are updated are used as the data balancing result of the server set.

[0147] In the embodiment of the present application, the peak-shaving and valley-filling principle is utilized to iteratively migrate the data of the Hadoop distributed file system from servers with a higher capacity share to servers with a lower capacity share in multiple rounds. Through this local balance and dynamic update of balance targets, an efficient global balance is achieved, which significantly improves the data balance of the Hadoop distributed file system, greatly reduces the difficulty and frequency of personnel maintenance, and also improves the resource utilization of the Hadoop cluster, thereby improving the overall performance and stability of the system.

[0148] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The server shown, such as Figure 6 As shown, it includes at least one processor 601 and a memory 602 connected to the at least one processor. The specific link medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0149] In the embodiment of the present application, the memory 602 stores instructions executed by at least one processor 601, and the at least one processor 601 can perform the steps of the above-mentioned data balance classification method by executing the instructions stored in the memory 602.

[0150] Among them, the processor 601 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and realize data balance by running or executing instructions stored in the memory 602 and calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing modules, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0151] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0152] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0153] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned data balancing method.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0155] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data balancing method, characterized in that: include: Perform multiple rounds of iterative data migration between servers in the server set until an iteration stop condition is met to obtain a data balance result of the server set, wherein each round of iteration includes the following operations: Obtaining a target server set including a migration-out server set and a migration-in server set from the server set; Obtaining the migration data volume of each migrated server in the migrated server set based on a first total data volume of the target server set and a first data difference volume between the target server set and a reference data volume corresponding to the server set in this iteration; According to the amount of migrated data of each of the migrated-out servers, the data of each of the migrated-out servers is migrated to the set of migrated-in servers, and the amount of data of each of the migrated-out servers is updated.

2. The method according to claim 1, characterized in that The step of obtaining a target server set including a migration-out server set and a migration-in server set from the server set includes: Sorting the multiple servers in the server set in ascending order of data volume to obtain a sorting result; Based on the ranking result, selecting the first M servers from the multiple servers to form the migration-in server set; and selecting the last L servers from the multiple servers to form the migration-out server set, wherein M and L are both greater than 1; The target server set is the union of the outgoing server set and the incoming server set.

3. The method according to claim 1, characterized in that The reference data volume corresponding to the server set in this round of iteration is obtained in the following way: Obtaining the average data volume of multiple servers in the server set during this round of iteration; The mean value of the data volume is multiplied by the number of target servers in the target server set to obtain a reference data volume corresponding to the server set in this round of iteration.

4. The method according to claim 1, characterized in that The step of obtaining the migration data volume of each migrated server in the migrated server set based on the first total data volume of the target server set and the first data difference volume between the target server set and the reference data volume corresponding to the server set in the current iteration includes: Based on the first data difference amount and a preset adjustment coefficient, obtaining the total amount of migrated data of the set of migrated servers; Based on the total amount of migrated data and the data migration ratio of each migrated-out server, the amount of migrated data of each migrated-out server is obtained.

5. The method according to claim 4, characterized in that The data migration ratio of each migrated server is obtained in the following manner: Obtaining a second data difference between the data volume of each migrated server and an average of the data volumes of multiple servers in the server set; Sum the second data difference amounts corresponding to a plurality of migrated servers in the migrated server set to obtain a total data difference amount; The ratio of the second data difference amount corresponding to each of the migrated-out servers to the total data difference amount is used as the data migration ratio of each of the migrated-out servers.

6. The method according to claim 1, characterized in that The step of migrating the data of each of the outgoing servers to the incoming server set according to the amount of migrated data of each of the outgoing servers includes: According to the amount of migrated data of each of the migrated servers and the amount of data of multiple migrated servers in the migrated server set, selecting at least one migrated server from the multiple migrated servers; The data of each of the outgoing servers is migrated to the at least one incoming server.

7. The method according to claim 6, characterized in that After migrating the data of each of the migrated-out servers to the at least one migrated-in server, the method further includes: The data volume of the at least one migration server is updated.

8. The method according to claim 7, characterized in that After the updating of the data volume of the at least one migrated server, the method further includes: Based on the updated data volumes of the multiple servers in the server set, obtaining a system performance indicator for this iteration; When the system performance index of the current iteration reaches a preset threshold, the iteration is stopped and the data volumes of the multiple servers in the server set after the current iteration are updated are used as the data balancing result of the server set.

9. A data balancing device, characterized in that: include: A processing module, used to perform multiple rounds of iterative data migration between servers in a server set until an iteration stop condition is met, and obtain a data balance result of the server set, wherein the processing module includes an acquisition module, a calculation module, and a migration module for executing each round of iteration; During each iteration: The acquisition module is used to obtain a target server set including a migration-out server set and a migration-in server set from the server set; The calculation module is used to obtain the migration data volume of each migrated server in the migrated server set based on the first total data volume of the target server set and the first data difference between the reference data volume corresponding to the server set in the current iteration; The migration module is used to migrate the data of each outgoing server to the incoming server set according to the amount of migrated data of each outgoing server, and to update the amount of data of each outgoing server.

10. A computer device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 8 according to the obtained program.

11. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer is caused to execute the method according to any one of claims 1 to 8.

Images