Balanced data migration method, device, equipment and storage medium

Abstract

The application relates to the field of data processing, in particular to a data migration balanced division method and device, equipment and a storage medium. The method comprises the following steps: acquiring a data set to be migrated and data sizes corresponding to n data in the data set; performing sorting processing on the data set according to the data sizes corresponding to the n data, to obtain a target set; performing division processing on the n data in the target set, to obtain m first multi-sets; performing summation processing on the data sizes in each first multi-set according to the data corresponding to the m first multi-sets, to obtain a target multi-set; performing variance calculation or standard deviation calculation on the target multi-set, to determine a first average value corresponding to the target multi-set; and according to the first average value and the data sizes corresponding to the n data, the n data are balancedly divided into m migration nodes. The method significantly improves the migration efficiency, and the data distribution of the nodes after capacity reduction is balanced enough.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a method, apparatus, device, and storage medium for balanced partitioning of data migration. Background Technology

[0002] With the rapid development of the Internet, distributed microservice architecture systems have become increasingly common. However, when a cluster architecture system is scaled down, in order to ensure that no data is lost, the plan is to migrate the data on the nodes that need to be scaled down to other nodes. This requires ensuring that the amount of data migrated to each machine is as balanced as possible, and the data migration is done in units of partitions.

[0003] Existing scaling-down methods generally employ allocation principles such as randomness, round-robin, or the average value algorithm (AVG). The data of the node that needs to be scaled down is randomly or in a round-robin manner allocated to other remaining nodes. The AVG algorithm takes a one-dimensional array as input, sorts the array in reverse order, calculates the average value of the array and denoted as AVG, and then traverses the array based on the AVG value and the AVG algorithm to obtain the required array.

[0004] However, the avg algorithm in the existing technology results in an uneven distribution of partitions, leading to problems such as low migration efficiency and uneven distribution of node data after scaling down. Summary of the Invention

[0005] This application provides a balanced partitioning method, apparatus, device, and storage medium for data migration to solve the aforementioned problems in the prior art, namely, that the avg algorithm in the prior art results in an unbalanced partition allocation, leading to low migration efficiency and unbalanced distribution of node data after scaling down.

[0006] Firstly, this application provides a balanced partitioning method for data migration, applied to an operations and maintenance platform, the method comprising:

[0007] Obtain the data set to be migrated and the data size corresponding to n data points in the data set;

[0008] Based on the size of the n data points, the data set is sorted to obtain the target set;

[0009] The n data points in the target set are partitioned to obtain m first multisets, where n ≥ m and m is an integer greater than 0;

[0010] Based on the data corresponding to the m first multisets, the data size within each first multiset is summed to obtain the target multiset, which includes the sum of the data sizes corresponding to each first multiset;

[0011] The variance or standard deviation of the target multiset is calculated to determine the first mean value corresponding to the target multiset.

[0012] Based on the first average value and the data size corresponding to the n data points, the n data points are evenly distributed among the m migration nodes.

[0013] Optionally, the target set includes n1, n2, n3...n n Where the data size corresponding to n1 is greater than the data size corresponding to n2, and the m first multisets include m1, m2, m3...m m The target multiset T includes t1, t2, t3...t m There is a correspondence between the first multiset and the target multiset. The step of calculating the variance or standard deviation of the target multiset to obtain the first average value corresponding to the target multiset includes:

[0014] The variance or standard deviation of the target multiset T is calculated using the following formula:

[0015]

[0016]

[0017]

[0018] Where, δ T 2 Let S be the variance of the target multiset T. T Let T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m.

[0019] Optionally, the m migration nodes include A1, A2, A3...A m The step of evenly distributing the n data points to m migration nodes based on the first average value and the corresponding data sizes of the n data points includes:

[0020] Balanced partitioning steps: Determine if n1 is less than the stated value. If not, then write n1 into m1 and evenly distribute the first multiset m1 to migration node A1;

[0021] When n1 is less than the When n1 is written into m1, the target set is traversed to select target data and written into m1, and the first multi-set m1 is evenly distributed to migration node A1, wherein the first multi-set m1 includes n1 and target data;

[0022] A second average value corresponding to the target multiset is determined, wherein the second average value is determined based on the new target set obtained by subtracting the first multiset from the target set;

[0023] Take the second average value as the new The balanced partitioning steps are repeated until the n data points are evenly distributed among the m migration nodes.

[0024] Optionally, iterating through the target set to select other data and writing it into m1 includes:

[0025] If the sum of n2 and t1 in the target set is greater than the Then continue traversing the target set to select n x ;

[0026] When n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value of n is given, the n value will be... x The n data is determined to be the target data. x Write to m1.

[0027] Optionally, the method further includes:

[0028] If the sum of n2 and t1 in the target set is less than the Continue iterating through the target set to select n x ;

[0029] When n2, n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value is n2 and n x The target data is determined, and n2 and n x Write to m1.

[0030] Secondly, this application provides a data migration balanced partitioning apparatus, the apparatus comprising:

[0031] Acquisition module: used to acquire the data set to be migrated and the data size corresponding to n data points in the data set;

[0032] The processing module is used to sort the data set according to the data size corresponding to the n data to obtain the target set;

[0033] The processing module is also used to partition the n data in the target set to obtain m first multisets, where n≥m and m is an integer >0;

[0034] The processing module is further configured to sum the data size in each of the m first multisets based on the data corresponding to the first multisets, to obtain a target multiset, wherein the target multiset includes the sum of the data size corresponding to each first multiset;

[0035] The processing module is further configured to perform variance calculation or standard deviation calculation on the target multiset to determine the first mean value corresponding to the target multiset;

[0036] The partitioning module is used to evenly distribute the n data points to m migration nodes based on the first average value and the data size corresponding to the n data points.

[0037] Optionally, the device further includes: a judgment module and a determination module;

[0038] The judgment module is used in the balanced partitioning step to determine whether n1 is less than the specified value. If not, then write n1 into m1 and evenly distribute the first multiset m1 to migration node A1;

[0039] The processing module is further configured to, when n1 is less than the When n1 is written into m1, the target set is traversed to select target data and written into m1, and the first multi-set m1 is evenly distributed to migration node A1, wherein the first multi-set m1 includes n1 and target data;

[0040] The determining module is used to determine the second average value corresponding to the target multiset, wherein the second average value is determined based on the new target set obtained by subtracting the first multiset from the target set;

[0041] The processing module is further configured to use the second average value as a new The balanced partitioning steps are repeated until the n data points are evenly distributed among the m migration nodes.

[0042] Optionally, the processing module is further configured to calculate the variance or standard deviation of the target multiset T using the following formula:

[0043]

[0044]

[0045]

[0046] Where, δ T 2 Let S be the variance of the target multiset T. TLet T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m.

[0047] Optionally, the processing module is further configured to, if the sum of n2 and t1 in the target set is greater than the sum of n2 and t1, then... Then continue traversing the target set to select n x ;

[0048] The processing module is also used when n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value of n is given, the n value will be... x The n data is determined to be the target data. x Write to m1.

[0049] Optionally, the processing module is further configured to, if the sum of n2 and t1 in the target set is less than the specified value, then... Continue iterating through the target set to select n x ;

[0050] The processing module is also used when n2, n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value is n2 and n x The target data is determined, and n2 and n x Write to m1.

[0051] Thirdly, this application provides a data migration balanced partitioning device, comprising: at least one processor and a memory;

[0052] The memory stores computer-executed instructions;

[0053] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform a balanced partitioning method for data migration as described in the first aspect and various possible implementations thereof.

[0054] Fourthly, embodiments of the present invention provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the balanced partitioning method for data migration described in the first aspect and various possible implementations of the first aspect.

[0055] This application provides a method, apparatus, device, and storage medium for balanced partitioning during data migration. The method involves: acquiring a data set to be migrated and the data sizes of n data points within that data set; sorting the data set according to the data sizes of the n data points to obtain a target set; partitioning the n data points in the target set to obtain m first multisets, where n ≥ m and m is an integer greater than 0; summing the data sizes within each of the m first multisets to obtain a target multiset, which includes the sum of the data sizes of each first multiset; calculating the variance or standard deviation of the target multiset to determine its first average; and evenly distributing the n data points across m migration nodes based on the first average and the data sizes of the n data points. This method solves the problem of uneven data distribution among nodes after rebalancing due to scattered and unbalanced data points, significantly improving migration efficiency and achieving a sufficiently balanced distribution of node data after scaling down. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 This application provides a flowchart of the balanced partitioning method for data migration. Figure 1 ;

[0058] Figure 2 This application provides a flowchart of the balanced partitioning method for data migration. Figure 2 ;

[0059] Figure 3 This is a schematic diagram of the data migration and balancing device provided in this application;

[0060] Figure 4 This is a schematic diagram of the data migration and balancing device provided in this application.

[0061] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0064] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] First, let me explain the terms used in this application:

[0066] avg algorithm: The avg algorithm in this application involves partitioning an array or set into a balanced form to obtain a relatively balanced target array or set. The following description uses an array as an example to illustrate this algorithm:

[0067] Step 1: Obtain a regular array, which may contain data of the same size, and sort the array in reverse order to obtain the target array;

[0068] Step 2: Calculate the average value (avg) for the target array;

[0069] Step 3: Iterate through the target array and perform corresponding processing. When the first element in the array is greater than or equal to the average, it is treated as a separate subarray, because adding more elements will not bring it closer to the average. Then, the remaining elements are redistributed evenly to make them more uniformly distributed, thus avoiding the influence of extreme values. Then, the next round of calculation begins.

[0070] When the first element is less than the average, it is added to the subarray. Then, one or more elements need to be found that make the sum of the current subarray closer to the target value. Here, the average is subtracted from the value of the current element, denoted as delta, and the array is traversed again. If an element is found whose value is exactly equal to delta, it is added to the subarray, and the search ends. If an element a is found that is greater than delta and less than another element b, the values ​​of (delta-b) and (a-delta) need to be compared. If (delta-b) is greater than (a-delta), b is added to the subarray, and delta is updated to delta-b. The array is then traversed again. If (delta-b) is less than (a-delta), the difference between b and delta is saved as distance, and a is added to the subarray. The search continues. If no more suitable element is found in subsequent traversals, b is added to the subarray, completing the second round of calculation.

[0071] The third round of calculation continues to traverse the array until it has been completely traversed and then evenly distributed to the corresponding subarrays, thus achieving a balanced distribution of the array.

[0072] Variance: Variance is a measure of the magnitude of fluctuation in a set of data. The smaller the variance, the smaller the fluctuation and the better the stability. The formula for calculating variance can be, for example:

[0073]

[0074] Where n is the number of data points in the set, t i Let i be the value of the i-th data in this set of data. Let i be the average value of the data set, where i is a positive integer ≤ n.

[0075] Standard deviation: The standard deviation is the square root of the variance. In a set of data, the larger the standard deviation, the more dispersed the distribution of its observations, and the worse its central tendency. Conversely, the smaller the standard deviation, the more concentrated the distribution of its observations, and the better its central tendency.

[0076] In existing technologies, when a distributed microservice architecture system scales down, in order to ensure that no data is lost, the plan is to migrate the data on the nodes that need to be scaled down to other nodes. This requires ensuring that the amount of data migrated to each machine is as balanced as possible. Existing scaling down methods generally adopt principles such as random, round-robin, or average value (AVG) algorithm to allocate the data of the nodes that need to be scaled down to other remaining nodes. The AVG algorithm takes a one-dimensional array as input, sorts the array in reverse order, calculates the average value of the array and records it as AVG, and traverses the array according to the AVG value and the AVG algorithm to obtain the required array.

[0077] The avg algorithm in current technology results in an uneven distribution of partitions, leading to problems such as low migration efficiency and uneven distribution of node data after scaling down.

[0078] To address the aforementioned issues, this application provides a balanced partitioning method for data migration. This method involves obtaining a set of n data points to be migrated, sorting these n data points to obtain a target set, partitioning the n data points in the target set into m first multisets, and summing the data within each of the m multisets to obtain the target multiset. The variance or standard deviation of the target multiset is then calculated. Based on the variance or standard deviation results and the size of the target set, the n data points are evenly distributed across the m migration nodes. This method solves the problem of uneven data distribution among nodes after rebalancing due to scattered and disorganized data points, significantly improving migration efficiency and achieving a sufficiently balanced distribution of node data after scaling down.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0080] Figure 1 This is a flowchart illustrating the balanced partitioning method for data migration provided in the embodiments of this application. Figure 1 .

[0081] like Figure 1 As shown in this embodiment, the data migration balanced partitioning method includes:

[0082] S101: Obtain the data set to be migrated and the data size corresponding to n data in the data set.

[0083] The data set to be migrated can be a single set or a multi-set, without specific restrictions here, and the set can contain n data elements.

[0084] A single set is a collection of data in which repeated elements are not allowed, and all elements are sorted. A multiset, on the other hand, allows repeated elements, and the same element can appear multiple times in a multiset.

[0085] S102: Sort the data set according to the data size corresponding to the n data to obtain the target set.

[0086] The sorting process for the target set is a reverse sorting process, and the target set may include, for example, n1, n2, n3...n n The data size of n1 is greater than the data size of n2.

[0087] S103: Divide the n data in the target set to obtain m first multisets.

[0088] Where n ≥ m, and m is an integer greater than 0. The n data points in the target set can be partitioned, for example, by dividing the n1, n2, and n3 data points in the target set into a multiset, and then dividing the remaining data into multiple multisets according to requirements. In this application, this multiset is referred to as the first multiset.

[0089] Understandably, if the n data points in the target set are divided into m multisets, then the m multisets correspond to the m first multisets in this application. These m first multisets can be, for example, m1, m2, m3...m m m1 may include, for example, n1, n2, and n3 of the first multiset mentioned above.

[0090] S104: Based on the data corresponding to the m first multisets, sum the data size within each first multiset to obtain the target multiset.

[0091] The target multiset includes the sum of the data sizes corresponding to each of the m first multisets. The data in one of the m first multisets can, for example, include n1, n2, n3. The summation of the data sizes in this first multiset can be, for example, n1 + n2 + n3. The data in the target multiset corresponding to this first multiset can be, for example, the sum of n1 + n2 + n3. The data included in the target multiset can, for example, be t1, t2, t3...t m In this target multiset, t1 can be, for example, the sum of n1+n2+n3.

[0092] Understandably, the determination of the corresponding data in the target multiset can be, for example, that the data corresponding to the m first multisets in the target multiset should also be m.

[0093] S105: Calculate the variance or standard deviation of the target multiset to determine the first mean value corresponding to the target multiset.

[0094] Among these, calculating the variance or standard deviation of the target multiset can be performed on, for example, on, t1, t2, t3...t in the target multiset. mThe variance or standard deviation is calculated. This calculation method can be implemented using existing techniques. The first mean corresponding to the target multiset can, for example, be the mean used in the variance or standard deviation calculation process.

[0095] Understandably, the first average value corresponding to the target set Ideally, this could be equal to the average of the m first multisets. In this application, the average of the first multisets represents the first average of the target set. Process it.

[0096] S106: Based on the first average value and the data size corresponding to the n data, evenly distribute the n data to the m migration nodes.

[0097] The m migration nodes can include, for example, A1, A2, A3...A m Let A1 be the first set of data evenly distributed among the n data points and m migration nodes. A method for evenly distributing the n data points among the m migration nodes could be as follows: To achieve a balanced data distribution, the difference between each data point corresponding to the m migration nodes during variance or standard deviation calculation and the mean of the target multiset corresponding to the n data points to be divided should be minimized. A way to minimize this difference is, for example, to combine the selection of each data point corresponding to the m migration nodes during variance or standard deviation calculation with the avg algorithm, thereby obtaining a more ideal balanced data distribution effect.

[0098] Optionally, the variance or standard deviation of the target multiset is calculated to obtain the first mean value corresponding to the target multiset. For example, the target set may include n1, n2, n3...n n Where the data size corresponding to n1 is greater than the data size corresponding to n2, and the m first multisets include m1, m2, m3...m m The target multiset T includes t1, t2, t3...t m There is a correspondence between the first multiset and the target multiset. The variance or standard deviation of the target multiset T is calculated using the following formula:

[0099]

[0100]

[0101]

[0102] Where, δT 2 Let S be the variance of the target multiset T. T Let T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m.

[0103] The balanced partitioning method for data migration provided in this embodiment obtains a set of n data points to be migrated, sorts these n data points to obtain a target set, partitions the n data points in the target set into m first multisets, sums the data within each of the m multisets to obtain a target multiset, calculates the variance or standard deviation of the target multiset, determines the first mean of the target multiset based on the variance or standard deviation calculation process, and then evenly partitions the n data points into m migration nodes based on the variance or standard deviation result and the data size of the target set. This method solves the problem of uneven data distribution among nodes after rebalancing due to scattered and non-concentrated data points, significantly improves migration efficiency, and ensures a sufficiently balanced distribution of node data after scaling down.

[0104] Figure 2 A flowchart illustrating the balanced partitioning method for data migration provided in this application embodiment. Figure 2 .like Figure 2 As shown, this embodiment is... Figure 1 Based on the embodiments, the balanced partitioning method for data migration is further explained. The balanced partitioning method for data migration shown in this embodiment includes:

[0105] S201: Obtain the data set to be migrated and the data size corresponding to n data in the data set.

[0106] Step S201 is similar to step S101 above, and will not be described in detail here.

[0107] S202: Sort the data set according to the size of the n data points to obtain a target set, wherein the target set includes n1, n2, n3...n n .

[0108] Step S202 is similar to step S102 above, and will not be described in detail here.

[0109] S203: Divide the n data points in the target set to obtain m first multisets, wherein the m first multisets include m1, m2, m3...m m .

[0110] Step S203 is similar to step S103 above, and will not be described in detail here.

[0111] S204: Based on the data corresponding to the m first multisets, sum the data size within each first multiset to obtain the target multiset, where the target multiset T includes t1, t2, t3...t m .

[0112] Step S204 is similar to step S104 above, and will not be described in detail here.

[0113] S205: Calculate the variance or standard deviation of the target multiset to determine the first mean value corresponding to the target multiset.

[0114] Step S205 is similar to step S105 above, and will not be described in detail here. The first average value is... This refers to t1, t2, t3...t in the target multiset in S105 above. m The first average used in the process of calculating variance or standard deviation, for example, can be...

[0115] S206: Determine whether n1 is less than the first average value.

[0116] If yes, proceed to step S207; otherwise, proceed to step S211.

[0117] Specifically, determining whether n1 is less than the first average value is equivalent to determining whether n1 is less than the first average value. The relationship between the data sizes.

[0118] Understandably, this step determines n1 and The size relationship between them is used to facilitate the balanced partitioning of data in the target set for different situations.

[0119] S207: Write n1 into m1. If the sum of n2 and t1 in the target set is greater than the first average value, continue to traverse the target set and select n. x .

[0120] This step refers to when n1 is less than the first average value. First, n1 is written into m1. Then, in order to make the sum of the data in the first multiset m1 corresponding to n1 closer to the sum of the sum of the sums ... Even if the data in the target multiset t1 corresponding to m1 is more evenly distributed to the corresponding migration node, which can be, for example, A1, then further processing of the data within the target set is needed. At this point, the value of t1 is tentatively set to n1. This processing can be achieved, for example, by summing n2 and t1 within the target set according to the avg algorithm. By comparison, if the sum of n2 and t1 is greater than... If this happens, it is necessary to iterate through the data in the target set again until data n in the target set is selected. x The sum of t1 and t1 equals Alternatively, once the data within the target set has been traversed, the current round of filtering is complete, meaning the selection of the migration node for this round is finished. This migration node can be, for example, the first migration node A1 among m migration nodes. If n2 and t1 are less than... At that time, the data within the target set can be processed in a way that, based on the principle of the avg algorithm, sums n2 and t1 within the target set. Comparisons are made, and data is then selected from the target set based on the comparison results.

[0121] S208: When n x When the sum of t1 and n is closest to the first average value, then n... x The n data is identified as the target data. x Write it into m1, and evenly distribute the first multiset m1 to migration node A1.

[0122] The first multiset m1 includes n1 and the target data. This step is equivalent to the above step where the sum of n2 and t1 is greater than... If this happens, it is necessary to iterate through the data in the target set again until data n in the target set is selected. x The sum of t1 and t1 equals Alternatively, once the data within the target set has been traversed, the current round of filtering is complete, and n... x Once the target data is identified and written into m1, the data in m1 will then be n. x And t1, evenly distribute the data in m1 to migration node A1, and complete the selection of migration nodes in this round.

[0123] Understandable, at this point n x The sum of t1 is closest to That is, at this time, the data dynamically selected from the target set determines that A1 among the m migration nodes is the migration node for the first round of balanced partitioning of that migration node.

[0124] This step is performed in parallel with step S210; both involve determining if the sum of n2 and t1 in the target set is greater than the first average value. If so, continue iterating through the target set to select n. x The specific execution steps for the two scenarios are determined based on the size of the data in the actual target set, and no specific restrictions are imposed here.

[0125] S209: If the sum of n2 and t1 in the target set is less than the first average value, continue to traverse the target set and select n. x .

[0126] First, n1 is written into m1. Then, in order to make the sum of the data in the first multiset m1 corresponding to n1 closer to the sum of the sum of the sums ... Even if the data in the target multiset t1 corresponding to m1 is more evenly distributed to the corresponding migration node, which could be A1, further processing of the data within the target set is required. In this case, the value of t1 is tentatively set to n1. If n2 is less than t1... When processing the data within the target set, for example, based on the principle of the avg algorithm, n2 is written into m1. Then, the data within the target set needs to be traversed and selected again until data n2 and n... within the target set are selected. x The sum closest to t1 and not greater than Alternatively, once the data within the target set has been traversed, the current round of filtering is complete, meaning the selection of the migration node for this round is finished. This migration node can be, for example, the first migration node A1 among m migration nodes; the data within the target set n2, n... x The sum of t1 and t1 is greater than If the sum of n2 and t1 in the target set is greater than the first average value, then similar processing shall be performed according to the case in step S207 above.

[0127] S210: When n2, n x When the sum of n2 and n1 is closest to the first average value, then n2 and n1 are... x The n2 and n are identified as target data. x Write it into m1, and evenly distribute the first multiset m1 to migration node A1.

[0128] The first multiset m1 includes n1 and the target data. This step is equivalent to the above statement: if the sum of n2 and t1 within the target set is less than the first average value... If this happens, it is necessary to iterate through the data in the target set again to select n. x Until data n within the target set is selected. x The sum of t1 and n2 equals Alternatively, once the data within the target set has been traversed, the current round of filtering is complete, and n2 and n... x Once the target data is identified and written into m1, the data in m1 will then be n. x Using n2 and t1, the data in m1 is evenly distributed to migration node A1, completing the selection of migration nodes for this round.

[0129] Understandable, at this point n x The sum of n2 and t1 is the closest. That is, at this time, the data dynamically selected from the target set determines that A1 among the m migration nodes is the migration node for the first round of balanced partitioning of that migration node.

[0130] S211: Write the n1 into the m1, and evenly distribute the first multiset m1 to the migration node A1.

[0131] Understandably, this step completes the selection of migration node A1.

[0132] S212: Determine the second average value corresponding to the target multiset.

[0133] The second average is determined by subtracting the first multiset from the target set to obtain the new target set. For example, the second average can be determined as follows: when the data in A1 is n1, the second average is the sum of the data in the target set minus n1, divided by m-1. In this case, the remaining data in the target set is n-1. When the data in A1 is n1 and n... x When the second average is the sum of the data in the target set minus n1 and n, x The value obtained is divided by m-1, at which point the target set has n-2 remaining data; the data in A1 are n1, n2, and n. x When the second average is the sum of the data in the target set minus n1, n2, and n... x The value obtained is divided by m-1. At this point, the target set has n-3 remaining data points. The data in A2 can be determined based on this second average value. This second average value can be denoted as... For the specific determination process, please refer to the extension of steps S207 to S210 above for determining the data of the next round of migration nodes.

[0134] Understandably, the determination of the third average value can be derived by analogy using this process as an example. Specifically, based on the data in A1 and A2, the third average value is the value obtained by dividing the sum of the data in the target set by the sum of the data in A1 and A2 by m-2. At this time, the remaining data in the target set is determined based on the number of data in A1 and A2. The data in A3 can be determined based on this third average value. For the specific determination process, please refer to the extension of the determination of the next round of migration node data in steps S207 to S210 above. The selection of the third round of migration node data is completed by analogy.

[0135] S213: Use the second average value as the new average value until the A is completed. m The partitioning of migration nodes evenly distributes n data points across m migration nodes.

[0136] This process involves repeating the process starting from the equalization partitioning step S207 until A is completed. m The partitioning of migration nodes evenly distributes n data points across m migration nodes.

[0137] Understandably, the data in A2 is determined based on this second average value. The specific determination process can be seen in steps S207-S210 above, which extend the determination of the first round of migration nodes. For example, the process of determining the data in A2 based on this second average value can be as follows:

[0138] When the data in the migration node A1 determined in the first round is n1, then the second average value... The value of m2 is obtained by subtracting n1 from the data in the target set and dividing the result by m-1. This dynamically selects data from the target set by subtracting n1 to determine the value of m2. Then, the value of t2 is determined based on the value of m2 until the data in the corresponding migration node A2 is determined. n2 is written into m2, and the value of t2 is temporarily set to n2. If the sum of n3 and t2 in the target set is greater than... Then continue traversing the target set to select n x When n x The sum of t2 is closest to t2. When n x The target data is determined, and n is set as the target data. x The data is written into m2, at which point the data in m2 is n. x And t2, evenly distribute the first multiset m2 to migration node A2; if the sum of n3 and t2 in the target set is less than Continue iterating through the target set to select n x When n3, n x The sum of t2 and t2 is closest. And not greater than When n3 and n x The n3 and n are identified as target data. x The data is written into m2, at which point the data in m2 are t2, n3, and n. x The first multiset m2 is evenly distributed to the migration node A2.

[0139] The data in the migration node A1 determined in the first round are n1 and n x When, then the second average value Subtract n1 and n from the sum of the data in the target set. x The value obtained by dividing the result by m-1 is the value obtained by dividing the result by m-1. At this point, the data in the target set is dynamically selected by subtracting n1 and n. xThe obtained data determines the value of m2, and then the value of t2 is determined based on the value of m2 until the data in the corresponding migration node A2 is determined. n2 is written into m2, and the value of t2 is temporarily set to n2. Here, it is assumed that the selected n... y Not equal to n3 or n x If the sum of n3 and t2 in the target set is greater than Then continue traversing the target set to select n y When n y The sum of t2 is closest to t2. When n y The target data is determined, and n is set as the target data. y The data is written into m2, at which point m2 contains t2 and n. y The first multiset m2 is evenly distributed to migration node A2; if the sum of n3 and t2 in the target set is less than Continue iterating through the target set to select n y When n3, n y The sum of t2 and t2 is closest. and not greater than When n3 and n y The n3 and n are identified as target data. y The data written into m2 are t2, n3, and n. y The first multiset m2 is evenly distributed to migration node A2, when n3, n y The sum of t2 and t2 is greater than Then, according to the above, if the sum of n3 and t2 in the target set is greater than... The same treatment should be applied to the situation at that time.

[0140] The data in the migration node A1 determined in the first round are n2, n3, and n x When, then the second average value Subtract n2, n3, and n from the sum of the data in the target set. x The value obtained by dividing the result by m-1 is the value obtained by dividing the result by m-1. At this point, the data in the target set is dynamically selected by subtracting n2, n3, and n... x The obtained data determines the value of m2, and then the value of t2 is determined based on the value of m2 until the data in the corresponding migration node A2 is determined. n4 is written into m2, and the value of t2 is temporarily set to n4. Here, it is assumed that the selected n... y Not equal to n5 or n x If the sum of n5 and t2 in the target set is greater than Then continue traversing the target set to select n y When n y The sum of t2 is closest to t2. When ny The target data is determined, and n is set as the target data. y The data is written into m2, and m2 contains t2 and n. y The first multiset m2 is evenly distributed to migration node A2; if the sum of n5 and t2 in the target set is less than Continue iterating through the target set to select n y When n5, n y The sum of t2 and t2 is closest. and not greater than At that time, n5 and n y The n5 and n were identified as target data, and the n5 and n y The data written into m2 are t2, n5, and n. y The first multiset m2 is evenly distributed to migration node A2, when n5, n y The sum of t2 and t2 is greater than Then, according to the above, if the sum of n5 and t2 in the target set is greater than... The same treatment should be applied to the situation at that time.

[0141] The determination of the third average value is then carried out by analogy using the process of the second round of migration nodes. Specifically, based on the data in A1 and A2, the third average value is the value obtained by dividing the sum of the data in the target set by the sum of the data in A1 and A2 by m-2. At this time, the remaining data in the target set is determined according to the number of data in A1 and A2. The data in A3 can be determined based on this third average value. For the specific determination process, please refer to the extension of the determination of the data for the next round of migration nodes in steps S207 to S210 above. The selection of the data for the third round of migration nodes is completed by analogy.

[0142] It is worth mentioning that n1, n2, n3...n involved in the above multiple embodiments n t1, t2, t3...t m , A1, A2, A3...A m All data are numbers greater than 0, which can be integers or decimals, depending on t1, t2, t3...t. m , A1, A2, A3...A m The corresponding n1, n2, n3...n n The actual content corresponding to the storage space is determined.

[0143] The data migration balanced partitioning method provided in this embodiment divides n data points evenly into m migration nodes based on the variance or standard deviation processing results and the data size of the target set. This is handled in two cases: when n1 is less than the first average, n1 is written into m1, and then two more cases are handled. The first case is that if the sum of n2 and t1 in the target set is greater than the first average, then the target set is traversed again to select n... x When n x When the sum of t1 and t1 is closest to the first average, n x The target data is determined, and n is set as the target data. x Write into m1, and evenly distribute the first multiset m1 to migration node A1. The second case is that if the sum of n2 and t1 in the target set is less than the first average, continue traversing the target set to select n. x , when n2, n x When the sum of n2 and n1 is closest to the first average, then n2 and n1 are... x The target data is determined, and n2 and n x Write n1 into m1, and evenly distribute the first multiset m1 to migration node A1, thus completing the data partitioning for this round when n1 is less than the first average value; when n1 is not less than the first average value, write n1 into m1, and evenly distribute the first multiset m1 to migration node A1, thus completing the data partitioning for this round when n1 is not less than the first average value.

[0144] After completing the first round of data partitioning—which divides n data points into m migration nodes based on variance or standard deviation and the size of the target set—the process proceeds by determining the second mean of the target multiset. This second mean is then used as the new mean, and the process of determining and dividing the data in both cases is repeated until the n data points are evenly distributed across the m migration nodes. This completes the balanced partitioning of the target set. This step, by processing the data in the target set in different cases, solves the problem of uneven data distribution after rebalancing due to scattered data points, significantly improving migration efficiency and ensuring a sufficiently balanced distribution of node data after scaling down.

[0145] Figure 3 This is a schematic diagram of the structure of a data migration and balanced partitioning device provided in an embodiment of this application. Figure 3 As shown, the data migration load balancing device 300 provided in this application includes:

[0146] Acquisition module 301: used to acquire the data set to be migrated and the data size corresponding to n data in the data set;

[0147] Processing module 302 is used to sort the data set according to the data size corresponding to the n data to obtain the target set;

[0148] The processing module 302 is further configured to partition the n data in the target set to obtain m first multisets, where n≥m and m is an integer >0;

[0149] The processing module 302 is further configured to sum the data size in each of the m first multisets based on the data corresponding to the first multisets, to obtain a target multiset, wherein the target multiset includes the sum of the data size corresponding to each first multiset;

[0150] The processing module 302 is further configured to perform variance calculation or standard deviation calculation on the target multiset to determine the first average value corresponding to the target multiset;

[0151] The partitioning module 303 is used to evenly partition the n data points into m migration nodes based on the first average value and the data size corresponding to the n data points.

[0152] Optionally, the device further includes: a judgment module 304 and a determination module 305;

[0153] The judgment module 304 is used in the balanced partitioning step to determine whether n1 is less than the specified value. If not, then write n1 into m1 and evenly distribute the first multiset m1 to migration node A1;

[0154] The processing module 302 is further configured to, when n1 is less than the When n1 is written into m1, the target set is traversed to select target data and written into m1, and the first multi-set m1 is evenly distributed to migration node A1, wherein the first multi-set m1 includes n1 and target data;

[0155] The determining module 305 is used to determine the second average value corresponding to the target multiset, wherein the second average value is determined based on the new target set obtained by subtracting the first multiset from the target set;

[0156] The processing module 302 is further configured to use the second average value as a new The balanced partitioning steps are repeated until the n data points are evenly distributed among the m migration nodes.

[0157] Optionally, the processing module 302 is further configured to calculate the variance or standard deviation of the target multiset T using the following formula:

[0158]

[0159]

[0160]

[0161] Where, δ T 2 Let S be the variance of the target multiset T. T Let T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m.

[0162] Optionally, the processing module 302 is further configured to, if the sum of n2 and t1 in the target set is greater than the sum of n2 and t1, then... Then continue iterating through the target set to select n x ;

[0163] The processing module 302 is further configured to, when n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value of n is given, the n value will be... x The n data is determined to be the target data. x Write to m1.

[0164] Optionally, the processing module 302 is further configured to, if the sum of n2 and t1 in the target set is less than the sum of n2 and t1, then... Continue iterating through the target set to select n x ;

[0165] The processing module 302 is further configured to handle n2, n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value is n2 and n x The target data is determined, and n2 and n x Write to m1.

[0166] Figure 4 A schematic diagram of the data migration load balancing device provided in this application. Figure 4 As shown, this application provides a data migration balancing partitioning device 400, which includes: a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0167] Receiver 401 is used to receive instructions and data;

[0168] Transmitter 402 is used to send commands and data;

[0169] Memory 404 is used to store instructions executed by the computer;

[0170] Processor 403 is used to execute computer execution instructions stored in memory 404 to implement the various steps of the data migration balanced partitioning method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the data migration balanced partitioning method.

[0171] Optionally, the memory 404 can be either standalone or integrated with the processor 403.

[0172] When the memory 404 is set up independently, the electronic device also includes a bus for connecting the memory 404 and the processor 403.

[0173] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data migration balanced partitioning method performed by the data migration balanced partitioning device described above.

[0174] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0175] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0176] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A balanced partitioning method for data migration, characterized in that, The method includes: Obtain the data set to be migrated and the data size corresponding to n data points in the data set; Based on the size of the n data points, the data set is sorted to obtain the target set; The n data points in the target set are partitioned to obtain m first multisets, where n ≥ m and m is an integer greater than 0; Based on the data corresponding to the m first multisets, the data size within each first multiset is summed to obtain the target multiset, which includes the sum of the data sizes corresponding to each first multiset; The variance or standard deviation of the target multiset is calculated to determine the first mean of the target multiset; the target set includes n1, n2, n3...n n Where the data size corresponding to n1 is greater than the data size corresponding to n2, and the m first multisets include m1, m2, m3...m m The target multiset T includes t1, t2, t3...t m There is a correspondence between the first multiset and the target multiset. The step of calculating the variance or standard deviation of the target multiset to obtain the first average value corresponding to the target multiset includes: The variance or standard deviation of the target multiset T is calculated using the following formula: ， ， in, Let t be the variance of the target multiset T. i Let i be the value of the i-th data in the target multiset. Let T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m; Based on the first average value and the data size corresponding to the n data points, the n data points are evenly distributed among the m migration nodes; The m migration nodes include A1, A2, A3...A m The step of evenly distributing the n data points to m migration nodes based on the first average value and the corresponding data sizes of the n data points includes: Balanced partitioning steps: Determine if n1 is less than the stated value. If not, then write n1 into m1 and evenly distribute the first multiset m1 to migration node A1; When n1 is less than the When n1 is written into m1, the target set is traversed to select target data and written into m1, and the first multi-set m1 is evenly distributed to migration node A1, wherein the first multi-set m1 includes n1 and target data; A second average value corresponding to the target multiset is determined, wherein the second average value is determined based on the new target set obtained by subtracting the first multiset from the target set; Take the second average value as the new The balanced partitioning steps are repeated until the n data points are evenly distributed among the m migration nodes.

2. The method according to claim 1, characterized in that, Traversing the target set to select other data and writing it into m1 includes: If the sum of n2 and t1 in the target set is greater than the Then continue traversing the target set to select n x ; When n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value of n is given, the n value will be... x The n data is determined to be the target data. x Write to m1.

3. The method according to claim 2, characterized in that, The method further includes: If the sum of n2 and t1 in the target set is less than the Continue iterating through the target set to select n x ; When n2, n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value is n2 and n x The target data is determined, and n2 and n x Write to m1.

4. A data migration balanced partitioning device, characterized in that, include: Acquisition module: used to acquire the data set to be migrated and the data size corresponding to n data points in the data set; The processing module is used to sort the data set according to the data size corresponding to the n data to obtain the target set; The processing module is also used to partition the n data in the target set to obtain m first multisets, where n≥m and m is an integer >0; The processing module is further configured to sum the data size in each of the m first multisets based on the data corresponding to the first multisets, to obtain a target multiset, wherein the target multiset includes the sum of the data size corresponding to each first multiset; The processing module is further configured to calculate the variance or standard deviation of the target multiset to determine the first mean value corresponding to the target multiset; the target set includes n1, n2, n3...n n Where the data size corresponding to n1 is greater than the data size corresponding to n2, and the m first multisets include m1, m2, m3...m m The target multiset T includes t1, t2, t3...t m There is a correspondence between the first multiset and the target multiset. The processing module is specifically used to calculate the variance or standard deviation of the target multiset T using the following formula: ， ， in, Let t be the variance of the target multiset T. i Let i be the value of the i-th data in the target multiset. Let T be the standard deviation of the target multiset T. Let i be the first average value of the target multiset T, where i is a positive integer ≤ m; The partitioning module is used to evenly distribute the n data points into m migration nodes based on the first average value and the data size corresponding to the n data points; The m migration nodes include A1, A2, A3...A m The judgment module is used in the balanced partitioning step to determine whether n1 is less than the specified value. If not, then write n1 into m1 and evenly distribute the first multiset m1 to migration node A1; The processing module is further configured to, when n1 is less than the When n1 is written into m1, the target set is traversed to select target data and written into m1, and the first multi-set m1 is evenly distributed to migration node A1, wherein the first multi-set m1 includes n1 and target data; The determining module is used to determine the second average value corresponding to the target multiset, wherein the second average value is determined based on the new target set obtained by subtracting the first multiset from the target set; The processing module is further configured to use the second average value as a new The balanced partitioning steps are repeated until the n data points are evenly distributed among the m migration nodes.

5. The apparatus according to claim 4, characterized in that, The processing module is further configured to, if the sum of n2 and t1 in the target set is greater than the sum of n2 and t1, then... Then continue traversing the target set to select n x ; The processing module is also used when n x The sum of t1 and t1 is closest to the sum of t1 and t1. When the value of n is given, the n value will be... x The n data is determined to be the target data. x Write to m1.

6. A data migration load balancing device, characterized in that, include: Memory; processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the balanced partitioning method for data migration as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the balanced partitioning method for data migration as described in any one of claims 1-3.

Images