Adaptive data equalization partitioning method, system, partitioner, and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a self-adaptive data equalization partitioning method, a self-adaptive data equalization partitioning system, a partitioner and a storage medium. The method comprises the following steps: counting the frequency of each key in all samples in intermediate data of a target stage, and adding the keys and corresponding samples to a to-be-allocated array; respectively cutting sample data according to initial allocation probabilities of partitions, and then allocating the sample data to different partitions, and updating the initial allocation probabilities of the partitions; and performing multi-round cutting and allocation on residual data according to the updated initial allocation probabilities, until the intermediate data are completely allocated, and a self-adaptive partitioning scheme is obtained. The application dynamically controls partitioning probabilities through a reward and punishment allocation strategy, and globally plans data allocation in combination with hot key cutting operation, so that the fundamental equalization of data allocation of the target stage is realized under the premise of low overhead.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an adaptive data balanced partitioning method and system, a partitioner and a computer readable storage medium. BACKGROUND

[0002] In a distributed computing scenario, Spark (a kind of distributed dataset) adopts HashPartitioner or RangePartitioner by default to divide the intermediate data in a Shuffle stage (a key process in distributed computing). However, when the data keys (Key) present a skewed distribution (such as a small number of keys associated with a large amount of data), the two partitioning methods will cause the partition data to be seriously imbalanced, resulting in significant increase in task execution time difference, decrease in cluster resource utilization and other problems.

[0003] The greedy strategy in the prior art can only guarantee a local optimal solution, and the global balanced effect of data partitioning is unstable; the feedback mechanism type technology significantly increases the communication cost and computing overhead between nodes, resulting in additional delay.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide an adaptive data balanced partitioning method and system, a partitioner and a computer readable storage medium, which aims to solve the problem of serious imbalance of partition data when partitioning and storing data in the prior art, thereby causing significant increase in task execution time difference and decrease in cluster resource utilization.

[0006] To achieve the above purpose, the present application provides an adaptive data balanced partitioning method, which comprises the following steps:

[0007] Sampling the intermediate data of a target stage to obtain a plurality of samples, counting the frequency of each key in all the samples, and adding all the keys and corresponding samples to a to-be-allocated array according to all the frequencies;

[0008] Initializing a plurality of partitions, cutting and allocating sample data of a plurality of samples in the to-be-allocated array to different partitions according to the initial allocation probability of each partition to obtain a plurality of partitioned samples containing residual data, and updating the initial allocation probability of each partition;

[0009] According to all the updated initial allocation probabilities, the plurality of partitioned samples are cut and allocated for multiple rounds until the intermediate data is completely allocated;

[0010] Convert all the allocation results of the intermediate data into a linked list form to obtain an adaptive partitioning scheme.

[0011] Optionally, the adaptive data equalization partitioning method, wherein the intermediate data of the target stage is sampled to obtain a plurality of samples, the frequency of each key in all the samples is counted, and all the keys and corresponding samples are added to the to-be-allocated array according to all the frequencies, specifically comprising:

[0012] Construct an initialized storage array, and store all key values of the Map task of the target stage into the storage array;

[0013] Traverse the intermediate data to store each sample in the intermediate data into the storage array, and construct a corresponding key-value pair according to each sample and each key value;

[0014] According to the proportion of each sample in the intermediate data, the key value corresponding to each sample is assigned a weight, and all the key-value pairs are sorted according to the weight of each key value to obtain a descending to-be-allocated array.

[0015] Optionally, the adaptive data equalization partitioning method, wherein the plurality of partitions are initialized, and the sample data of a plurality of samples in the to-be-allocated array are respectively cut and allocated into different partitions according to the initial allocation probability of each partition to obtain a plurality of partitioned samples containing residual data, and the initial allocation probability of each partition is updated, specifically comprising:

[0016] Construct a plurality of partitions, and initialize the existing data amount of all the partitions to 0;

[0017] Initialize the residual capacity of all the partitions to the maximum storable data amount:

[0018] ;

[0019] ;

[0020] wherein, represents the index of the partition, represents the existing data amount of the i-th partition, represents the residual capacity of the i-th partition, represents the maximum capacity of the partition, represents the storable tolerance of the partition; According to the number of all the partitions, the initial allocation probability of each partition is defined:

[0021] According to the number of all the partitions, the initial allocation probability of each partition is defined:

[0022] ​ ;

[0023] wherein, represents the initial allocation probability of the first partitions, represents the number of all partitions;

[0024] According to the initial allocation probability of each of the partitions, the samples corresponding to the plurality of key-value pairs in the to-be-allocated array are matched with different partitions;

[0025] If the data amount of the sample is not greater than the remaining capacity of the partition, the sample is stored into the corresponding partition; if the data amount of the sample is greater than the remaining capacity of the partition, the sample is cut and then allocated into the corresponding partition to obtain a split sample containing residual data.

[0026] The weights of all the split samples are updated, and the initial allocation probability of each of the partitions is updated to obtain a secondary allocation probability.

[0027] Optionally, the adaptive data balancing partitioning method, wherein if the data amount of the sample is not greater than the remaining capacity of the partition, the sample is stored into the corresponding partition; if the data amount of the sample is greater than the remaining capacity of the partition, the sample is cut and then allocated into the corresponding partition to obtain a split sample containing residual data, specifically comprises:

[0028] According to the order of all the key-value pairs in the to-be-allocated array, the relationship between the data amount of the sample corresponding to each of the key-value pairs and the remaining capacity of the corresponding partition is determined according to the weight of each of the key-value pairs.

[0029] If the data amount of the current sample is not greater than the remaining capacity of the partition, the current sample is stored into the partition in whole, and the remaining capacity of the corresponding partition is updated.

[0030] If the data amount of the current sample is not greater than the remaining capacity of the partition, the current sample is cut, the data exceeding the remaining capacity of the partition is cut into a split sample containing residual data, and the remaining capacity of the partition is updated to 0.

[0031] Optionally, the adaptive data balancing partitioning method, wherein the weights of all the split samples are updated, and the initial allocation probability of each of the partitions is updated to obtain a secondary allocation probability, specifically comprises:

[0032] Based on the proportion of each remaining data to all remaining data, the weight of each remaining data is obtained, and all remaining data are input into the array to be allocated in descending order according to the weight, to obtain a descending sorted secondary array to be allocated;

[0033] The initial allocation probability of all partitions with a remaining capacity of 0 is updated to 0, and all partitions with a remaining capacity of non-zero are defined as multiple secondary partitions;

[0034] Based on the difference between the amount of data stored in each of the secondary partitions and the average amount of data stored in all partitions, the initial allocation probability of each secondary partition is updated to obtain the corresponding secondary allocation probability:

[0035] ;

[0036] ;

[0037] ;

[0038] in, Indicates the first The difference between the amount of data in each secondary partition and the average amount of data in all partitions. Indicates the first The penalty range for each secondary partition This represents the average amount of data stored across all partitions. Indicates the first The amount of existing data in each partition. Indicates the penalty coefficient. This indicates the amount of data currently present in all partitions. Indicates the first The probability of secondary allocation for each partition.

[0039] Optionally, the adaptive data balancing partitioning method, wherein the step of performing multiple rounds of cutting and allocating all the segmented samples according to all updated initial allocation probabilities until all the intermediate data has been allocated, specifically includes:

[0040] Based on the secondary allocation probability of each secondary partition, all segmented samples in the secondary array to be allocated are stored in each secondary partition;

[0041] If the current data volume of the segmented sample is not greater than the remaining capacity of the secondary partition, then all the current segmented samples are stored in the secondary partition, and the remaining capacity of the corresponding secondary partition is updated.

[0042] if the data amount of the split sample is greater than the remaining capacity of the secondary partition, splitting the split sample to obtain first data and second data of different sizes from the remaining capacity, and storing the first data in the secondary partition;

[0043] defining all secondary partitions with remaining capacity as a plurality of tertiary partitions, updating the weight of all the second data, sorting all the second data according to all the weights, and storing all the second data in all the tertiary partitions respectively until the intermediate data is completely allocated.

[0044] Optionally, the adaptive data balanced partitioning method, wherein the intermediate data is converted into a linked list form to obtain an adaptive partitioning scheme, specifically comprising:

[0045] creating a key-value mapping dictionary, and adding the key value of each target sample to the key-value mapping dictionary, wherein the target sample includes the sample of the intermediate data and the split sample after splitting;

[0046] adding all target samples with the same key value to the same linked list according to the key-value mapping dictionary to obtain a plurality of tuples;

[0047] wherein each of the tuples includes all partition indexes of the sample and the weight corresponding to each partition index and / or all partition indexes of the split sample and the weight corresponding to each partition index;

[0048] calculating the total weight of all the tuples, and sorting all the tuples according to all the total weights to obtain an adaptive partitioning scheme in the form of a linked list.

[0049] In addition, to achieve the above-mentioned purposes, the application further provides an adaptive data balanced partitioning system, wherein the adaptive data balanced partitioning system comprises:

[0050] a sampling module for sampling the intermediate data of a target stage to obtain a plurality of samples, counting the frequency of each key in all the samples, and adding all the keys and corresponding samples to a to-be-allocated array according to all the frequencies;

[0051] a first storage module for initializing a plurality of partitions, cutting and allocating sample data of a plurality of samples in the to-be-allocated array to different partitions according to the initial allocation probability of each partition to obtain a plurality of split samples containing remaining data, and updating the initial allocation probability of each partition;

[0052] The second storage module is configured to filter a plurality of secondary partitions from all the partitions according to all the updated initial allocation probabilities, and to respectively allocate a plurality of the remaining data to different secondary partitions after cutting the plurality of the remaining data according to the capacity of each of the secondary partitions, until all the intermediate data is completely allocated.

[0053] The linked list conversion module is configured to convert the allocation results of all the partitions and all the secondary partitions into a linked list form to obtain an adaptive partitioning scheme.

[0054] In addition, to achieve the above object, the application further provides a partitioner, which comprises a memory, a processor, and an adaptive data equalization partitioning program stored in the memory and executable on the processor, and the adaptive data equalization partitioning program implements the steps of the adaptive data equalization partitioning method when executed by the processor.

[0055] In addition, to achieve the above object, the application further provides a computer readable storage medium, which stores an adaptive data equalization partitioning program, and the adaptive data equalization partitioning program implements the steps of the adaptive data equalization partitioning method when executed by a processor.

[0056] In the application, the intermediate data of a target stage is sampled to obtain a plurality of samples, the frequency of each key in all the samples is counted, all the keys and corresponding samples are added to a to-be-allocated array according to all the frequencies, a plurality of partitions are initialized, sample data of a plurality of the samples in the to-be-allocated array are respectively allocated to different partitions after cutting according to the initial allocation probability of each of the partitions to obtain a plurality of partitioned samples containing remaining data, and the initial allocation probability of each of the partitions is updated, all the partitioned samples are cut and allocated for multiple rounds according to all the updated initial allocation probabilities until the intermediate data is completely allocated, and all the allocation results of the intermediate data are converted into a linked list form to obtain an adaptive partitioning scheme. The application dynamically regulates the partition probability through a reward and punishment allocation strategy, globally coordinates data allocation in combination with a hot key cutting operation, and realizes the fundamental equalization of data allocation of the target stage under the premise of low overhead. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of a preferred embodiment of the adaptive data equalization partitioning method of the application;

[0058] Figure 2 is a structural schematic diagram of a partitioner of a preferred embodiment of the adaptive data equalization partitioning method of the application;

[0059] Figure 3is a partition flow chart of a preferred embodiment of the adaptive data equalization partition method of the present application;

[0060] Figure 4 is a structure diagram of a preferred embodiment of the adaptive data equalization partition system of the present application;

[0061] Figure 5 is a structure diagram of a preferred embodiment of the partitioner of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0063] The adaptive data equalization partition method described in the preferred embodiment of the present application, as shown in Figure 1 The adaptive data equalization partition method comprises the following steps:

[0064] Step S10, sampling the intermediate data of the target stage to obtain a plurality of samples, counting the frequency of each key in all the samples, and adding all the keys and corresponding samples to the to-be-allocated array according to all the frequencies.

[0065] In a distributed computing scenario, Spark (a kind of distributed dataset) adopts hash partitioning or range partitioning by default to divide the intermediate data of the Shuffle stage (i.e. the target stage); however, the existing two partitioning methods have their own problems: for the greedy partitioning strategy, the core is to estimate the key weight by sampling, to fill the partition bucket in descending order of data volume, and to perform splitting operation on the keys exceeding the bucket capacity, which can reduce the overload of a single partition, but excessively relies on the sampling accuracy, and the greedy strategy only guarantees local optimization, and the global equalization effect is unstable; for the feedback mechanism type technology, a learning automaton is deployed at each node, and through the iteration process of multiple rounds of behavior selection-feedback-probability update, the data allocation strategy is dynamically adjusted, but it relies on multiple rounds of feedback iteration, significantly increases the inter-node communication cost and computing overhead (such as probability vector update), and causes additional delay.

[0066] In order to overcome the above problems, the present application significantly improves the efficiency, resource utilization and task execution stability of distributed computing through an innovative adaptive data equalization partitioner. Specifically, the adaptive data equalization partitioner focuses on the problem of uneven partition data caused by key distribution tilt (such as excessive data volume of a few hot keys) in the Shuffle phase of the Spark framework. By combining the reward and punishment allocation strategy with the hot key segmentation operation, the following key defects are solved from the root cause, and the adaptive mechanism is realized to fundamentally balance the Spark data partition under the premise of low additional overhead, thereby reducing the task running time by an average of 8.12%-29.90%, and improving the cluster resource utilization to more than 95%.

[0067] Specifically, an initialized storage array is constructed, and all key values of the target stage Map task are stored in the storage array; the intermediate data is traversed to store each sample in the intermediate data into the storage array, and a corresponding key-value pair is constructed according to each sample and each key value; a weight is assigned to the key value corresponding to each sample according to the proportion of each sample in the intermediate data, and all key-value pairs are sorted according to the weight of each key value to obtain a descending to-be-allocated array.

[0068] As shown in the formula (1), the adaptive data equalization partitioner realizes the partition storage of the intermediate data of the Shuffle phase through the cooperation of the four parts, and the core principle is to dynamically control the partition probability through the reward and punishment allocation strategy, and to globally plan the data allocation combined with the hot key segmentation operation, so as to realize the fundamental balance of the Shuffle phase data under the premise of low overhead. Figure 2

[0069] Firstly, in the sampling and weight estimation part, the intermediate data of the Shuffle phase is sampled through the reservoir algorithm, and these samples are stored in the initial storage array (an array that has not been sorted), and then the frequency of key occurrence of each sample is counted, so as to inversely deduce the weight of each key corresponding to the sample in all data according to the sampling ratio (the weight of the sample represents the proportion of the data amount of the sample in all data), thereby obtaining the key-value pair (i.e. the combination of weight and key) of each sample, wherein the probability of each data being selected is constant and unbiased estimation, so this process ensures the unbiasedness of weight estimation; then each sample is sorted according to the key-value pair (also can be through weight or key), thereby obtaining a descending to-be-allocated array.

[0070] Through the above sampling process, for the adaptive streaming big data scenario, the continuously generated Shuffle intermediate data can be dynamically processed, and only a single traversal is required to generate unbiased samples, avoiding repetition and ensuring low memory overhead and high stability, supporting accurate execution of the subsequent reward and punishment strategy.​

[0071] Step S20: Initialize multiple partitions. Based on the initial allocation probability of each partition, the sample data of multiple samples in the array to be allocated are cut and allocated to different partitions to obtain multiple segmented samples containing the remaining data, and the initial allocation probability of each partition is updated.

[0072] In the process of initializing each partition before allocation, the initial maximum capacity of each partition is the same, so the initial allocation probability of each partition is the same. Based on the initial allocation probability, the samples in the array to be allocated are stored into different partitions one by one, thereby achieving adaptive allocation.

[0073] Specifically, multiple partitions are constructed, and the existing data volume of all partitions is initialized to 0; the remaining capacity of all partitions is initialized to the maximum amount of data that can be stored.

[0074] ;

[0075] ;

[0076] in, Indexes representing partitions, Indicates the first The amount of existing data in each partition. Indicates the first The remaining capacity of each partition, Indicates the maximum capacity of the partition. This represents the storability tolerance of a partition; based on the total number of all said partitions, the initial allocation probability of each said partition is defined:

[0077] ;

[0078] in, Indicates the first The initial allocation probability of each partition. The number of partitions is represented; based on the initial allocation probability of each partition, multiple key-value pairs in the array to be allocated are matched with different partitions; if the data volume of a sample is not greater than the remaining capacity of the partition, the sample is stored in the corresponding partition; if the data volume of a sample is greater than the remaining capacity of the partition, the sample is split and allocated to the corresponding partition to obtain the corresponding segmented sample containing the remaining data; the weights of all the segmented samples are updated, and the initial allocation probability of each partition is updated to obtain the corresponding secondary allocation probability.

[0079] Among them, such as Figure 3As shown, the data of all samples in the to-be-allocated array (in descending order) is stored in different partitions one by one. If the data amount of a sample is greater than the maximum capacity of a partition, the data of the sample is split, the data of the current sample is split into the same data amount as the maximum capacity of the partition, and the split remaining data is stored in the current partition. The split remaining data is stored in the original position of the to-be-allocated array for the next round of data partition storage. Since the current partition is filled, the maximum capacity of the current partition becomes 0, and the allocation probability of the current partition also becomes 0. Thus, in the subsequent rounds of data partition storage, other samples will not store data in this partition.

[0080] Further, according to the order of all the key-value pairs in the to-be-allocated array, the relationship between the data amount of each key-value pair and the remaining capacity of the corresponding partition is determined according to the weight of each key-value pair. If the data amount of the current sample is not greater than the remaining capacity of the partition, the current sample is stored in the partition, and the remaining capacity of the corresponding partition is updated. If the data amount of the current sample is not greater than the remaining capacity of the partition, the current sample is split, and the data exceeding the remaining capacity of the partition is split into a split sample containing remaining data, and the remaining capacity of the partition is updated to 0.

[0081] In the first round of data partition storage, the data amount of multiple samples is greater than the maximum capacity of the partition, so corresponding secondary samples are split and stored in the to-be-allocated array. For samples arranged later, the data amount is less than the maximum capacity of the partition, so they can be stored in the corresponding partition. At this time, these partitions have different remaining capacities, and the secondary allocation probability of each partition can be updated according to the remaining capacity of each partition. Since all samples in the to-be-allocated array before allocation are arranged in descending order according to the weight ratio, after the first allocation is performed, all secondary samples in the to-be-allocated array are still arranged in descending order according to the weight, and the to-be-allocated array after the first allocation does not need to be reordered.

[0082] Further, the weight of each remaining data is obtained according to the proportion of each remaining data in all remaining data, and all the remaining data is input into the to-be-allocated array in descending order according to the weight, to obtain a to-be-allocated secondary array arranged in descending order. The initial allocation probability of all partitions with a remaining capacity of 0 is updated to 0, and all partitions with a remaining capacity other than 0 are defined as multiple secondary partitions. The initial allocation probability of each secondary partition is updated according to the difference between the amount of data stored in each secondary partition and the average amount of data stored in all partitions, to obtain the corresponding secondary allocation probability.

[0083] ;

[0084] ;

[0085] ;

[0086] in, Indicates the first The difference between the amount of data in each secondary partition and the average amount of data in all partitions. Indicates the first The penalty range for each secondary partition This represents the average amount of data stored across all partitions. Indicates the first The amount of existing data in each partition. Indicates the penalty coefficient. This indicates the amount of data currently present in all partitions. Indicates the first The probability of secondary allocation for each partition.

[0087] The partitioning scheme is updated by allocating the remaining keys in the ordered array according to the reward and penalty allocation strategy and hotkey splitting operation. The reward and penalty function refers to a calculation method that adjusts the allocation probability of each partition through rewards or penalties, including reward and penalty coefficients, reward and penalty formulas, and reward and penalty results. In the embodiments disclosed in this invention, the existing data volume of each partition, the total existing data volume, the average data volume of each partition, and the total data volume of all keys are used as parameters of the reward and penalty formula, which, together with the reward and penalty coefficients, adjust the reward and penalty results to increase or decrease the allocation probability of each partition. The data-balanced partitioning effect is best when all partitions are allocated the same amount of data, i.e., all data is evenly distributed among the partitions. At this time, under the same conditions, the system's running time for the corresponding work unit of each partition is the shortest. In the adaptive partitioning method, the partition data volume under this ideal state is used as the ideal capacity of each partition, and also as one of the conditions for subsequently distinguishing the states of each partition.

[0088] When the amount of data already in a partition exceeds the average amount of data already in the partition, continuing to allocate data to that partition may cause or exacerbate the data skew between partitions. Conversely, when the amount of data already in a partition is far less than the average amount of data already in the partition, allocating the data to be partitioned to that partition is beneficial in reducing the data volume gap between that partition and other heavily loaded partitions, i.e., reducing the variance of the existing data volume of each partition. Therefore, the adaptive partitioning method uses the average amount of data already in the current partition as the judgment condition and adjustment factor of the reward and penalty function.

[0089] Wherein, after the first allocation is completed, the next allocation is theoretically not suitable for allocating data to the partition, and the allocation probability of the partition needs to be appropriately punished. The more data that has been allocated, or the greater the mean difference of the existing data volume of the partition, the less suitable it is to allocate, and the greater the punishment needs to be. Conversely, the smaller the punishment, and the corresponding change in the secondary allocation probability of the partition, i.e. updating the partition that can allocate data (i.e. the partition with remaining capacity not equal to 0) to a secondary partition.

[0090] Step S30, according to all the updated initial allocation probabilities, perform multiple rounds of cutting and allocating on all the segmented samples until all the intermediate data is allocated.

[0091] Wherein, for the to-be-allocated array after the first allocation, there may be some samples whose data volume is still greater than the remaining capacity of the corresponding secondary partition, so these samples still need to be segmented, and after the remaining capacity of the secondary partition is filled, the remaining data segmented out is stored in the corresponding to-be-allocated array. When the second round of data partition storage is completed, the weight order of all samples in the to-be-allocated array may no longer be in descending order, so all samples in the to-be-allocated array need to be rearranged to obtain a descending array.

[0092] Specifically, according to the secondary allocation probability of each secondary partition, all segmented samples in the to-be-allocated secondary array are respectively stored in each secondary partition; if the data volume of the current segmented sample is not greater than the remaining capacity of the secondary partition, the current segmented sample is stored in the secondary partition, and the remaining capacity of the corresponding secondary partition is updated; if the data volume of the segmented sample is greater than the remaining capacity of the secondary partition, a first data of the same size as the remaining capacity and a second data of different size are segmented from the current segmented sample, and the first data is stored in the secondary partition; all secondary partitions with remaining capacity are defined as multiple tertiary partitions, the weight of all second data is updated, and after all the second data is sorted according to all the weights, all the second data is respectively stored in all tertiary partitions until all the intermediate data is allocated.

[0093] The data amount of the first secondary sample (i.e. the remaining data after the first sample is divided) is the largest for the first round of partition storage of the to-be-allocated array, and the data amount of the last sample in the first round of partition storage is the smallest (the weight is the smallest), so in this round, the initial allocation probability of the corresponding storage partition is updated, and the allocation probability of the secondary partition is the largest, so in the second round, the first secondary sample is stored in the secondary partition with the largest probability, and if the data amount of the first secondary sample is greater than the remaining capacity of the secondary partition, the first secondary sample still needs to be divided, and the divided data is stored in the corresponding position of the to-be-allocated array.

[0094] When the second round of data partition storage is completed, the samples in the to-be-allocated array may no longer be arranged in descending order, so the weights (i.e. the proportion of the data amount of each sample in the total remaining data) of the samples need to be updated and the samples need to be arranged in descending order again. Each allocated secondary partition continues to update the allocation probability according to the remaining capacity according to the reward and punishment allocation strategy, and the secondary partition that can continue to store data is updated to a tertiary partition, and the steps described above are performed until all the data is completed.

[0095] The adaptive partitioning method described in the application selects appropriate partitions for the data of each key by combining the reward and punishment allocation strategy and the hot key division operation, rewards or punishes the allocation probability of each partition, divides or completely allocates the data of the key, and adaptively balances the data amount allocated to each partition, so as to finally obtain an adaptive partitioning scheme in which the data amounts of each partition are relatively balanced. The reward and punishment allocation strategy and the hot key division operation are not completely independent, but are alternately executed. The general process of this module is that the reward and punishment allocation strategy updates the allocation probability of each partition in real time according to the reward and punishment function, selects the partition with the highest allocation probability as the intended partition each time, then determines whether to perform the hot key division operation according to the size relationship between the remaining capacity of the intended partition and the data amount of the key, and repeats the above steps until all the data of all the keys in the sample is allocated.

[0096] Step S40, converting all allocation results of the intermediate data into a linked list form to obtain an adaptive partitioning scheme.

[0097] In the partition scheme obtained by the adaptive partition module, due to the hot key division operation, the same key may be allocated to different partitions. In order to facilitate the identification of the subsequent actual partition process, in the partition scheme summary conversion module, the elements of the same key in the original partition scheme are summarized, the cumulative proportion of the data amount allocated to different partitions in the same key is calculated, and the chain table form is converted, the node element is the combination of the partition and the corresponding cumulative proportion, and the final adaptive partition scheme is obtained. At this time, the cumulative proportion of each node in the chain table and the difference value of the previous node indicate the data proportion of the corresponding data of the key allocated to the partition pointed to by the node. The larger the difference value, the more data of the key allocated to the partition.

[0098] Specifically, a key-value mapping dictionary is created, all allocation results of each round of allocation are traversed, and the key value of each target sample is added to the key-value mapping dictionary, wherein the target sample includes the sample of the intermediate data and the segmented sample; according to the key-value mapping dictionary, all target samples with the same key value are added to the same chain table to obtain multiple tuples; wherein each tuple includes all partition indexes of the sample and a weight corresponding to each partition index and / or all partition indexes of the segmented sample and a weight corresponding to each partition index; the total weight of all tuples is calculated; all the tuples are sorted according to all the total weights, and an adaptive partition scheme in the form of a chain table is obtained.

[0099] Among them, due to the hot key division operation, the data of the same key (such as the high-frequency word key) may be allocated to multiple partitions (for example, part of the data of key K1 is in partition 1, and another part is in partition 2). The original partition scheme (scheme) is a discrete item (key, partition number, weight), which is difficult to efficiently process the subsequent actual partition. The core of the conversion is to convert the original partition scheme (key-partition-weight triple) into a chain table form through traversal, grouping, and cumulative calculation.

[0100] First, a dictionary is initialized, the key (key) is a string type (representing the key name), and the value is a chain table type (storing the partition and weight pair); for example, in the original scheme, the key K1 is allocated to partition 1 (weight 100) and partition 2 (weight 50), and the key-value mapping dictionary is initialized to chain table [(partition=1, weight=100), (partition=2, weight=50)] (wherein, partition represents the partition number of sample storage, and weight represents the sample weight).

[0101] Then traverse all the storage schemes of each round, add all (partition, weight) pairs of the same key to the corresponding linked list; it should be noted that the linked list of each key is a sequential list, and the node element is a tuple (partition, weight), and the weight represents the amount of data allocated to the partition (for example, the linked list of hot key K1 may be [((1, 100), (2, 50))], multiple allocation records of the same key are combined into the same linked list, ensuring data integrity.

[0102] Further, the total weight (wsum) is calculated, and the cumulative ratio (ratio) is calculated. For each key's linked list, calculate the weight sum, then traverse the linked list, calculate the cumulative ratio of each node, and build a new linked list to store (partition, cumulative ratio).

[0103] For example: the total weight of key K1 is 150: [((1, 100), (2, 50))].

[0104] First node: ratio = 0 + (100 / 150): node value = (1, 0.666).

[0105] Second node: ratio = 0.666 + (50 / 150): node value = (2, 1.0).

[0106] Difference meaning: the first node difference (0.666-0) represents that 66.6% of the data is allocated to partition 1; the second node difference (1.0-0.666) represents that 33.3% of the data is allocated to partition 2.

[0107] Among them, the conversion process can be realized in the following way:

[0108] "Input: partition scheme ;

[0109] Output: summarized partition scheme ;

[0110] Steps:

[0111] Create , key is a string type, value is a linked list type;

[0112] FOR DO;

[0113] IF already contains DO;

[0114] ;

[0115] ELSE;

[0116] Create a new key in the middle ;

[0117] ;

[0118] END IF;

[0119] END FOR;

[0120] FOR DO;

[0121] ;

[0122] FOR DO;

[0123] ;

[0124] ;

[0125] END FOR;

[0126] END FOR”。

[0127] When all the nodes of the linked list are traversed, that is, the final allocation scheme (adaptive partitioning scheme) is generated, and the actual partitioning operation is performed according to the final allocation scheme. In the actual partitioning, each data of the Shuffle stage is allocated to a partition according to the converted adaptive partitioning scheme. For data containing keys not included in the partitioning scheme, the data is allocated to a partition according to the hash partitioning method. For data containing keys included in the partitioning scheme, first, the linked list corresponding to the key in the partitioning scheme is obtained, and a random number in the range of (0, 1) is generated, and the random number is compared with the cumulative proportion value in the linked list node in turn. When the random number is less than the proportion value of the accessed node, the data is allocated to the partition pointed to by the node, otherwise the next node of the linked list is accessed. The difference between the proportion of the current accessed node and the last node represents the probability of the current data being allocated to the partition pointed to by the accessed node. Through the random number with equal probability, the data of the same key in the partitioning scheme is divided into different partitions according to different data volume proportions.

[0128] This invention proposes an adaptive data balancing partitioning method, designing and implementing a corresponding partitioner. It achieves data balancing partitioning during the Shuffle stage by combining a reward-penalty allocation strategy with hotkey splitting operations. This method uses a reward-penalty function to adjust the allocation probability of each partition. After each allocation, it increases or decreases the probability of each partition in the next allocation based on its state, adaptively selecting the most suitable partition for each key. Simultaneously, for keys exceeding the remaining capacity of a partition, the method automatically divides the key's data into multiple parts based on the remaining capacity, and allocates each part to an appropriate partition based on the allocation probability. This method effectively mitigates key skew and its adverse effects in cases of unbalanced key distribution, achieving relatively efficient balancing partitioning for data with varying degrees of skew, thus improving Spark's performance and resource utilization.

[0129] Furthermore, such as Figure 4 As shown, based on the above-described adaptive data balancing partitioning method, the present invention also provides an adaptive data balancing partitioning system, wherein the adaptive data balancing partitioning system includes:

[0130] The sampling module 51 is used to sample the intermediate data of the target stage to obtain multiple samples, count the frequency of each key in all the samples, and add all the keys and corresponding samples to the array to be assigned according to all the frequencies.

[0131] The first storage module 52 is used to initialize multiple partitions, and according to the initial allocation probability of each partition, to cut the sample data of multiple samples in the array to be allocated and allocate them to different partitions to obtain multiple segmented samples containing the remaining data, and to update the initial allocation probability of each partition.

[0132] The second storage module 53 is used to filter multiple secondary partitions from all the partitions according to all the updated initial allocation probabilities, and according to the capacity of each secondary partition, to cut the remaining data and allocate them to different secondary partitions until all the intermediate data is allocated.

[0133] The linked list conversion module 54 is used to convert the allocation results of all the partitions and all the secondary partitions into linked list form to obtain an adaptive partitioning scheme.

[0134] Furthermore, such as Figure 5 As shown, based on the above-described adaptive data balancing partitioning method and system, the present invention also provides a partitioner, which includes a processor 10, a memory 20, and a display 30. Figure 5 Only a portion of the partitioner's components are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead.

[0135] The memory 20 can be an internal storage unit of the partitioner in some embodiments, such as a hard disk or a memory of the partitioner. The memory 20 can also be an external storage device of the partitioner in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 can include both an internal storage unit and an external storage device of the partitioner. The memory 20 is used to store application software and various data installed on the partitioner, such as program codes of the installed partitioner, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores an adaptive data balancing partition program 40, which can be executed by the processor 10 to implement the adaptive data balancing partition method.

[0136] The processor 10 can be a Central Processing Unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the adaptive data balancing partition method, etc.

[0137] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the partitioner and to display a visualized user interface. The components of the partitioner communicate with each other through a system bus.

[0138] In an embodiment, the processor 10 implements the steps of the adaptive data balancing partition method as described above when executing the adaptive data balancing partition program 40 in the memory 20.

[0139] The application also provides a computer readable storage medium, which stores an adaptive data balancing partition program. When the adaptive data balancing partition program is executed by a processor, the steps of the adaptive data balancing partition method as described above are implemented.

[0140] In summary, this invention provides an adaptive data balancing partitioning method and related equipment. The method includes: sampling intermediate data in the Shuffle stage to obtain multiple samples; counting the frequency of each key in all samples; adding all keys and corresponding samples to a to-be-allocated array according to all frequencies; initializing multiple partitions; according to the initial allocation probability of each partition, splitting the sample data of the multiple samples in the to-be-allocated array and allocating them to different partitions to obtain multiple split samples containing remaining data; updating the initial allocation probability of each partition; performing multiple rounds of splitting and allocation on all the split samples according to all updated initial allocation probabilities until all intermediate data is allocated; converting all allocation results of the intermediate data into a linked list to obtain an adaptive partitioning scheme. This invention dynamically adjusts partition probabilities through a reward and punishment allocation strategy, and combines hotkey splitting operations to globally coordinate data allocation, achieving fundamental balance in the allocation of data in the Shuffle stage with low overhead.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or partitioner that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or partitioner. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or partitioner that includes that element.

[0142] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0143] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An adaptive data balancing partitioning method, characterized in that, The adaptive data balancing partitioning method includes: The intermediate data of the target stage is sampled to obtain multiple samples. The frequency of each key in all the samples is counted. All the keys and their corresponding samples are added to the array to be assigned based on all the frequencies. Multiple partitions are initialized. Based on the initial allocation probability of each partition, the sample data of multiple samples in the array to be allocated are split and allocated to different partitions to obtain multiple segmented samples containing the remaining data. The initial allocation probability of each partition is then updated. The initialization of multiple partitions, based on the initial allocation probability of each partition, involves splitting the sample data of multiple samples in the array to be allocated into different partitions, resulting in multiple segmented samples containing remaining data, and updating the initial allocation probability of each partition. Specifically, this includes: Create multiple partitions and initialize the existing data volume of all partitions to 0; Initialize the remaining capacity of all partitions to the maximum amount of data that can be stored: ； ； in, Indexes representing partitions, Indicates the first The amount of existing data in each partition. Indicates the first The remaining capacity of each partition, Indicates the maximum capacity of the partition. Indicates the storability tolerance of the partition; Based on the total number of all said partitions, define the initial allocation probability for each said partition: ； in, Indicates the first The initial allocation probability of each partition. Indicates the number of all partitions; Based on the initial allocation probability of each partition, the samples corresponding to multiple key-value pairs in the array to be allocated are matched with different partitions; If the amount of data in a sample is not greater than the remaining capacity of the partition, the sample is stored in the corresponding partition. If the amount of data in a sample is greater than the remaining capacity of the partition, the sample is split and allocated to the corresponding partition to obtain the corresponding segmented sample containing the remaining data. Update the weights of all the segmented samples and update the initial allocation probability of each partition to obtain the corresponding secondary allocation probability; All the segmented samples are cut and assigned in multiple rounds based on all updated initial assignment probabilities until all the intermediate data has been assigned. All allocation results of the intermediate data are converted into linked list form to obtain an adaptive partitioning scheme.

2. The adaptive data balancing partitioning method according to claim 1, characterized in that, The process of sampling intermediate data in the target stage to obtain multiple samples, counting the frequency of each key in all samples, and adding all keys and corresponding samples to the unassigned array based on all frequencies specifically includes: Construct an initial storage array and store all key-value pairs of the Map task in the target phase into the storage array; The intermediate data is traversed, and each sample in the intermediate data is stored in the storage array. Based on each sample and each key value, a corresponding key-value pair is constructed. Based on the proportion of each sample in the intermediate data, a weight is assigned to the key value corresponding to each sample, and all key value pairs are sorted according to the weight of each key value to obtain a descending array to be assigned.

3. The adaptive data balancing partitioning method according to claim 1, characterized in that, If the data volume of a sample is not greater than the remaining capacity of the partition, the sample is stored in the corresponding partition. If the data volume of a sample is greater than the remaining capacity of the partition, the sample is split and allocated to the corresponding partition to obtain a corresponding segmented sample containing the remaining data. Specifically, this includes: According to the order of all the key-value pairs in the array to be allocated, and based on the weight of each key-value pair, determine the relationship between the data volume of the corresponding sample and the remaining capacity of the corresponding partition. If the current data volume of the sample is not greater than the remaining capacity of the partition, then all the current samples are stored in the partition, and the remaining capacity of the corresponding partition is updated. If the current sample data volume is not greater than the remaining capacity of the partition, then the current sample is segmented, the data exceeding the remaining capacity of the partition is segmented into segmented samples containing the remaining data, and the remaining capacity of the partition is updated to 0.

4. The adaptive data balancing partitioning method according to claim 3, characterized in that, The step of updating the weights of all the segmented samples and updating the initial allocation probability of each partition to obtain the corresponding secondary allocation probability specifically includes: Based on the proportion of each remaining data to all remaining data, the weight of each remaining data is obtained, and all remaining data are input into the array to be allocated in descending order according to the weight, to obtain a descending sorted secondary array to be allocated; The initial allocation probability of all partitions with a remaining capacity of 0 is updated to 0, and all partitions with a remaining capacity of non-zero are defined as multiple secondary partitions; Based on the difference between the amount of data stored in each of the secondary partitions and the average amount of data stored in all partitions, the initial allocation probability of each secondary partition is updated to obtain the corresponding secondary allocation probability: ； ； ； in, Indicates the first The difference between the amount of data in each secondary partition and the average amount of data in all partitions. Indicates the first The penalty range for each secondary partition This represents the average amount of data stored across all partitions. Indicates the first The amount of existing data in each partition. Indicates the penalty coefficient. This indicates the amount of data currently present in all partitions. Indicates the first The probability of secondary allocation for each partition.

5. The adaptive data balancing partitioning method according to claim 4, characterized in that, The step of performing multiple rounds of cutting and allocation on all the segmented samples based on all updated initial allocation probabilities until all intermediate data has been allocated specifically includes: Based on the secondary allocation probability of each secondary partition, all segmented samples in the secondary array to be allocated are stored in each secondary partition; If the current data volume of the segmented sample is not greater than the remaining capacity of the secondary partition, then all the current segmented samples are stored in the secondary partition, and the remaining capacity of the corresponding secondary partition is updated. If the amount of data in the segmented sample is greater than the remaining capacity of the secondary partition, then a first data with the same size as the remaining capacity and a second data with a different size are segmented from the current segmented sample, and the first data is stored in the secondary partition; Define all secondary partitions with remaining capacity as multiple tertiary partitions, update the weights of all the second data, sort all the second data according to all the weights, and then re-store all the second data into all the tertiary partitions until all the intermediate data is allocated.

6. The adaptive data balancing partitioning method according to claim 1, characterized in that, The step of converting all allocation results of the intermediate data into a linked list to obtain an adaptive partitioning scheme specifically includes: Create a key-value mapping dictionary, iterate through all allocation results in each round of allocation, and add the key value of each target sample to the key-value mapping dictionary, wherein the target sample includes the sample in the intermediate data and the segmented sample after segmentation; Based on the key-value mapping dictionary, all target samples with the same key value are added to the same linked list to obtain multiple tuples; Each tuple includes all partition indices of the sample and the weight corresponding to each partition index and / or all partition indices of the segmented sample and the weight corresponding to each partition index; Calculate the total weight of all the tuples and sort all the tuples according to the total weight to obtain an adaptive partitioning scheme in the form of a linked list.

7. An adaptive data balancing partitioning system, characterized in that, The adaptive data balancing partitioning system is used to implement the adaptive data balancing partitioning method according to any one of claims 1-6, including: The sampling module is used to sample the intermediate data of the target stage to obtain multiple samples, count the frequency of each key in all the samples, and add all the keys and corresponding samples to the array to be assigned according to all the frequencies. The first storage module is used to initialize multiple partitions, and according to the initial allocation probability of each partition, to cut the sample data of multiple samples in the array to be allocated and allocate them to different partitions to obtain multiple segmented samples containing the remaining data, and to update the initial allocation probability of each partition. The second storage module is used to filter multiple secondary partitions from all the partitions according to all the updated initial allocation probabilities, and to divide the remaining data into different secondary partitions according to the capacity of each secondary partition, until all the intermediate data is allocated. The linked list conversion module is used to convert the allocation results of all the partitions and all the secondary partitions into linked list form to obtain an adaptive partitioning scheme.

8. A partitioner, characterized in that, The partitioner includes: a memory, a processor, and an adaptive data balancing partitioning program stored in the memory and executable on the processor, wherein the adaptive data balancing partitioning program, when executed by the processor, implements the steps of the adaptive data balancing partitioning method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an adaptive data balancing partitioning program, which, when executed by a processor, implements the steps of the adaptive data balancing partitioning method as described in any one of claims 1-6.

Images