Intermediate data partitioning method and system based on data locality and load balancing perception in Spark framework

Abstract

The invention discloses an intermediate data partitioning method based on data locality and load balancing perception in a Spark framework, which comprises the following steps of: firstly, counting the distribution condition of each key cluster on each node of a Spark cluster aiming at a large amount of intermediate key value pair data generated in a Shuffle stage, and constructing a distribution inclination model capable of quantifying the data imbalance degree among the nodes; therefore, more real and comprehensive data distribution information can be obtained; then, classifying all the key clusters by using a distribution gradient index, and dividing the key clusters into inclined distribution and non-inclined distribution so as to realize differentiated distribution strategy design; then, for the key clusters with inclined distribution, a concept of preference nodes is introduced, and a plurality of key clusters which are densely distributed on the same preference node are preferentially distributed into the same partition, so that the local centrality of data is enhanced, and the data locality optimization of a reduce task is realized; and finally, on the basis of the preliminary partitioning result, a load balancing distribution strategy is adopted for the key clusters which are distributed in a non-inclined mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data and distributed parallel computing, and more specifically, relates to a method and system for partitioning intermediate data based on data locality and load balancing awareness in a Spark framework. Background Art

[0002] The Spark framework is a big data processing framework based on an in-memory computing model, widely used in scenarios such as iterative computing, interactive queries, and stream processing. During Spark's execution, raw input data and intermediate processing results are typically organized into partitions. Each partition is processed independently by a task, and multiple tasks are executed in parallel across multiple nodes in the cluster. A typical Spark job consists of multiple stages: the map stage (the first stage) and the reduce stage (all subsequent stages). The map stage is responsible for generating intermediate results from the input data, while the reduce stage processes the intermediate key / value pairs generated by the previous stage's tasks to complete the final computation. When there are wide dependencies between the map and reduce stages, or between reduce and reduce stages, Spark must perform a shuffle operation. This means that the reduce task processing a partition must pull its portion of intermediate data from each node to ensure that each reduce task receives all the data it needs. Therefore, when the amount of intermediate data distributed to different partitions is significantly uneven, the reduce task load can vary significantly, a phenomenon known as data skew. This can cause some tasks to take an unusually long time to execute, lowering the overall efficiency of the job. Furthermore, the intermediate data partitioning method determines the distribution of key-value pairs across partitions, which in turn affects the effectiveness of reduce tasks in optimizing execution using data locality. Reduce tasks are prioritized for scheduling to nodes that store the largest amount of data for the partitions they process, minimizing cross-node data transmission and reducing network communication overhead. Therefore, a reasonable data partitioning method is crucial in the Spark framework. It not only affects task parallelism and data locality, but also effectively alleviates problems such as uneven computing load and communication bottlenecks, directly determining the overall execution efficiency and resource utilization of the job.

[0003] Currently, the Spark framework supports two primary data partitioning methods by default. The first is a hash-based data partitioning method. This method evenly distributes key-value pairs across partitions by hashing the key and then taking the modulus of the number of partitions. This strategy is simple to implement and suitable for scenarios with a relatively balanced key-value distribution. The second is a range-based data partitioning method. This method obtains global key ordering information through sampling and divides the key into several contiguous intervals, thus achieving order between partitions. This method is commonly used in operations such as sorting and window aggregation.

[0004] However, all of these methods have some significant drawbacks: First, hash-based data partitioning assumes a uniform key distribution, but data skew often occurs in real-world data, leading to overloaded partitions and impacting overall job performance. Second, range-based data partitioning can mitigate data skew to a certain extent, but its partition boundaries rely on the representativeness of the sampled data and are susceptible to sample bias, resulting in uneven data loads in some partitions. Third, neither of these data partitioning methods considers the impact of data locality on reduce tasks, limiting the effectiveness of network communication optimization for these tasks. Summary of the Invention

[0005] In response to the above defects or improvement needs of the existing technology, the present invention provides an intermediate data partitioning method and system based on data locality and load balancing awareness in the Spark framework, which aims to solve the technical problem that the existing Hash-based data partitioning method often suffers from data skew in real data, resulting in excessive load on some partitions and affecting the overall job performance; and the existing Range-based data partitioning method has a technical problem that the partition boundaries depend on the representativeness of the sampled data and are easily affected by sample bias, resulting in uneven data volume in some partitions; and both of the above two data partitioning methods lack data locality awareness of the reduce task, resulting in the need to transmit a large amount of intermediate data through the network during the shuffle process, thereby bringing about a high communication overhead problem.

[0006] To achieve the above object, according to one aspect of the present invention, a method for partitioning intermediate data based on data locality and load balancing awareness in a Spark framework is provided, comprising the following steps: (1) Obtain the Spark jobs in the Spark cluster and the intermediate data stored in each node, and classify all the intermediate data according to their corresponding keys to obtain a key cluster set , where n is the total number of key categories, that is, the total number of key clusters in the key cluster set, represents the key cluster corresponding to the key of the i-th category, and i∈[1,n]; (2) Obtain each key cluster from the key cluster set obtained in step (1) Corresponding key The set of key-value pairs distributed across all nodes in the Spark cluster.

[0007] (3) Each bond cluster obtained in step (2) Corresponding key The number of key-value pairs distributed on each node in the Spark cluster is collected to obtain the distribution skewness index of the key cluster. .

[0008] (4) The distribution skewness index of each bond cluster obtained in step (3) With preset parameters , get all bond clusters with skewed distribution and all bond clusters with non-skewed distribution; (5) Prioritize all key clusters with skewed distribution obtained in step (4) to obtain a preliminary partition set.

[0009] (6) Based on the preliminary partition set obtained in step (5), load balancing is performed on all key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

[0010] Preferably, the preset parameters in step (4) satisfy , if the distribution skewness index of a key cluster Greater than parameter , then the key cluster is a key cluster with skew distribution. If the distribution skewness index of a key cluster is Less than or equal to the parameter , then the bond cluster is a bond cluster with non-skewed distribution.

[0011] Preferably, step (2) comprises the following sub-steps: (2-1) Get the rack set consisting of all racks in the Spark cluster And all nodes in all racks, all nodes in all racks constitute the node set in Indicates the size of the rack set, Represents the size of the node set, and has (2-2) The node set obtained in step (2-1) Get each node Stored intermediate data , the intermediate data stored by all nodes constitute the intermediate data set ,in ∈[1, ], and there is: ; in Indicates the nodes Stored intermediate data The i-th key in Display key The corresponding j-th value, Indicates the nodes Stored intermediate data The middle button is , the value is Key-value pairs, For intermediate data The total number of keys included in , and there is ; (2-3) Get each key based on the intermediate data set obtained in step (2-2) At each node The corresponding key-value pair distribution number , this key The number of key-value pairs on all nodes constitutes the key-value pair distribution quantity set .

[0012] Preferably, step (3) comprises the following sub-steps: (3-1) Determine whether there is a map-side combination operation triggered by a typical aggregation operator (such as reduceByKey, combineByKey, etc.) in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3); (3-2) The change ratio of the number of key-value pairs before and after the map-side merge operation For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Update to get the size of the key cluster , then go to step (3-4); (3-3) For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Perform statistical processing to obtain the size of the key cluster , then go to step (3-4); (3-4) Based on the size of each bond cluster obtained Get the key-value pair distribution number set of the key corresponding to the key cluster on all nodes Average value and standard deviation ; (3-5) The set of key-value pairs corresponding to each key cluster obtained in step (3-4) on each node Average value and standard deviation Get the distribution skewness index of the key cluster ; Preferably, step (5) includes the following sub-steps: (5-1) Obtain a set of bond clusters consisting of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a partition set consisting of multiple partitions from the Spark job obtained in step (3-1) ,in Indicates the total number of partitions, Indicates the partitions; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preference nodes ,in Indicates the preference nodes; (5-4) Set counter cnt1=1; (5-5) Determine whether cnt1 is greater than If yes, go to step (5-10), otherwise go to step (5-6); (5-6) Determine the cnt1th key cluster obtained in step (5-3) Is the total size less than or equal to the average partition size? If yes, go to step (5-7), otherwise go to step (5-8); (5-7) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) A partition that is not assigned a key cluster is obtained to obtain a partition that is assigned a key cluster, and then go to steps (5-9); (5-8) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) Multiple partitions that are not assigned key clusters are obtained to obtain multiple partitions that are assigned key clusters, and then go to steps (5-9); (5-9) Set counter cnt1 = cnt1 + 1 and return to step (5-5); (5-10) Returns the partition set consisting of all partitions that have been assigned the key cluster as the preliminary partition set.

[0013] Preferably, the total size of a key cluster group refers to the sum of the sizes of all key clusters in the key cluster group.

[0014] Average partition size Equal to the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

[0015] Preferably, steps (5-8) include the following sub-steps: (A1) Group the bond clusters All key clusters in are sorted in descending order according to the key cluster size to obtain the sorted key cluster group ,in Represents a key cluster group the total number of mid-key clusters; (A2) Set counter cnt2=1; (A3) Determine whether cnt2 is greater than the key cluster group The total number of key clusters If yes, go to step (A9), otherwise go to step (A4); (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether there is a partition , meeting the following two conditions: 1) its current partition size is the largest in the partition set; 2) the partition After being assigned a key cluster, its current partition size does not exceed the average partition size If yes, go to step (A6), otherwise go to step (A7), where y represents the sequence number of the partition that meets the condition in the partition set; (A6) Cluster the bonds Assign to partition , to get an updated partition , then go to step (A8); (A7) Cluster the bonds Assigned to the partition set obtained in step (5-2) A partition that has not yet been assigned a key cluster to obtain a partition that has been assigned a key cluster Then go to step (A8); (A8) Set counter cnt2 = cnt2 + 1 and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions assigned key clusters.

[0016] Preferably, step (6) includes the following sub-steps: (6-1) Sort all key clusters with non-skewed distribution obtained in step (4) in descending order according to the size of the key cluster to obtain multiple sorted key clusters ,in represents the total number of bond clusters with non-skewed distribution obtained in step (4); (6-2) Set counter cnt3 = 1; (6-3) Determine whether cnt3 is greater than If yes, go to step (6-6), otherwise go to step (6-4); (6-4) Select the smallest partition from the partition set obtained in step (5-2) and add the cnt3th key cluster from the multiple key clusters sorted in step (6-1) to the partition set. Put it into the partition to obtain the updated partition, and then go to step (6-5); (6-5) Set counter cnt3 = cnt3 + 1 and return to step (6-3); (6-6) Return all updated partitions as the final partition set.

[0017] According to another aspect of the present invention, an intermediate data partitioning system based on data locality and load balancing awareness in a Spark framework is provided, comprising: The first module is used to obtain the Spark jobs in the Spark cluster and the intermediate data stored in each node, and classify all the intermediate data according to their corresponding keys to obtain a key cluster set. , where n is the total number of key categories, that is, the total number of key clusters in the key cluster set, represents the key cluster corresponding to the key of the i-th category, and i∈[1,n]; The second module is used to obtain each key cluster according to the key cluster set obtained by the first module. Corresponding key The set of key-value pairs distributed across all nodes in the Spark cluster.

[0018] The third module is used for each key cluster obtained from the second module Corresponding key The number of key-value pairs distributed on each node in the Spark cluster is collected to obtain the distribution skewness index of the key cluster. .

[0019] The fourth module is used to calculate the distribution tilt index of each key cluster obtained from the third module. With preset parameters , get all bond clusters with skewed distribution and all bond clusters with non-skewed distribution; The fifth module is used to preferentially allocate all key clusters with skewed distribution obtained by the fourth module to obtain a preliminary partition set.

[0020] The sixth module is used to perform load balancing on all key clusters with non-skewed distribution obtained in the fourth module based on the preliminary partition set obtained in the fifth module to obtain a final partition set.

[0021] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art: (1) The present invention adopts steps (1) to (4), and by introducing a key cluster distribution skewness modeling method, it can distinguish the data distribution patterns of different key clusters between nodes, thereby identifying key clusters with obvious skew characteristics, and realizing explicit modeling of the non-uniformity of intermediate data between nodes; therefore, it can effectively solve the technical problem that the existing Hash-based data partitioning method assumes uniform key distribution and is difficult to deal with the uneven load caused by data skew; (2) Due to the adoption of step (4), the present invention makes full use of the global real distribution information when classifying key clusters, rather than relying on sampling data for division, and performs inclination calculation and classification based on the actual distribution; therefore, it can effectively solve the technical problems of the existing range-based partitioning method that is highly dependent on sampling results and is easily affected by sample bias, thereby improving the accuracy and robustness of partition boundary division; (3) The present invention introduces a priority allocation strategy in step (5). For key clusters with skewed distribution, multiple key clusters distributed on the same preference node are allocated to the same partition as much as possible to enhance the data concentration of the partition on the node, thereby maximizing the data locality of the reduce task, reducing cross-node data transmission, and effectively reducing the network communication overhead and scheduling delay in the Shuffle process; therefore, it can effectively solve the problem of high communication overhead caused by the lack of data locality awareness of the reduce task in the existing Hash-based data partitioning method and Range-based partitioning method, which results in a large amount of cross-node transmission of intermediate data in the Shuffle process.

[0022] (4) In step (6), the present invention proposes a load balancing allocation strategy for non-skewed distributed key clusters, and dynamically adjusts the allocation of remaining key clusters based on the existing initial partitioning results to achieve overall balance at the partition granularity, thereby improving the efficiency of parallel task execution and ensuring the stability and scalability of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0025] The basic idea of ​​the present invention is to jointly optimize the partitioning effect of Spark intermediate data from four aspects. First, for the large amount of intermediate key-value pair data generated in the Shuffle stage, the distribution of each key cluster on each node of the Spark cluster is statistically analyzed, and a distribution skew model that can quantify the degree of data imbalance between nodes is constructed, thereby obtaining more real and comprehensive data distribution information; secondly, all key clusters are classified using the distribution skew index and divided into two categories: skewed distribution and non-skewed distribution, so as to achieve differentiated allocation strategy design. Then, for key clusters with skewed distribution, the concept of preference nodes is introduced, and multiple key clusters densely distributed on the same preference node are preferentially allocated to the same partition, thereby enhancing the local concentration of data and achieving data locality optimization of reduce tasks. Finally, based on the preliminary partitioning results, a load balancing distribution strategy is adopted for key clusters with non-skewed distribution to balance the total data volume of each partition, improve the efficiency of parallel execution, and ultimately significantly reduce the network communication overhead and performance bottlenecks caused by data skew while ensuring the stability of job execution.

[0026] First, some technical terms of the present invention are explained and illustrated below: Key cluster: refers to the collection of all key-value pairs with the same key.

[0027] Preferred node: For a key cluster, its preferred node is defined as the node that stores the largest number of key / value pairs corresponding to the key cluster among all nodes.

[0028] like Figure 1 As shown, the present invention provides an intermediate data partitioning method based on data locality and load balancing awareness in a Spark framework, comprising the following steps: (1) Obtain the Spark jobs in the Spark cluster and the intermediate data (key-value pairs) stored in each node, and classify all the intermediate data according to their corresponding keys to obtain a key cluster set. , where n is the total number of key categories, that is, the total number of key clusters in the key cluster set, Represents the key cluster corresponding to the key of the i-th category, and i∈[1,n].

[0029] (2) Obtain each key cluster from the key cluster set obtained in step (1) Corresponding key The set of key-value pairs distributed across all nodes in the Spark cluster.

[0030] (3) Each bond cluster obtained in step (2) Corresponding key The number of key-value pairs distributed on each node in the Spark cluster is collected to obtain the distribution skewness index of the key cluster. .

[0031] (4) The distribution skewness index of each bond cluster obtained in step (3) With preset parameters , get all bond clusters with skewed distribution and all bond clusters with non-skewed distribution; Specifically, the preset parameters , if the distribution skewness index of a key cluster Greater than parameter , then the key cluster is a key cluster with skew distribution. If the distribution skewness index of a key cluster is Less than or equal to the parameter , then the bond cluster is a bond cluster with non-skewed distribution.

[0032] The advantage of this step (4) is that it achieves the classification of key clusters, making the entire partitioning process more adaptable and dynamic. By decoupling the skewed and non-skewed data, the system's versatility and robustness in multiple scenarios and data characteristics are improved.

[0033] (5) Prioritize all key clusters with skewed distribution obtained in step (4) to obtain a preliminary partition set.

[0034] The advantage of this step (5) is that it adopts a priority allocation strategy for key clusters with skewed distribution, which not only ensures data locality but also avoids excessive load on a single partition, thus achieving an effective balance between partition load balancing and data skew mitigation.

[0035] (6) Based on the preliminary partition set obtained in step (5), load balancing is performed on all key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

[0036] The advantage of the above steps (1) to (6) is that they systematically combine data locality and load balancing awareness, and implement an efficient intermediate data partition allocation strategy by differentially processing skewed and non-skewed key clusters, effectively alleviating the data skew problem and improving the execution efficiency and resource utilization of Spark jobs.

[0037] Specifically, step (2) includes the following sub-steps: (2-1) Get the rack set consisting of all racks in the Spark cluster And all nodes in all racks, all nodes in all racks constitute the node set in Indicates the size of the rack set, Represents the size of the node set, and has (2-2) The node set obtained in step (2-1) Get each node Stored intermediate data , the intermediate data stored by all nodes constitute the intermediate data set ,in ∈[1, ], and there is: ; in Indicates the nodes Stored intermediate data The i-th key in Display key The corresponding j-th value, Indicates the nodes Stored intermediate data The middle button is , the value is Key-value pairs, For intermediate data The total number of keys included in , and there is ; (2-3) Get each key based on the intermediate data set obtained in step (2-2) At each node The corresponding key-value pair distribution number , this key The number of key-value pairs on all nodes constitutes the key-value pair distribution quantity set .

[0038] Specifically, step (3) includes the following sub-steps: (3-1) Determine whether there is a map-side combination operation triggered by a typical aggregation operator (such as reduceByKey, combineByKey, etc.) in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3); (3-2) The change ratio of the number of key-value pairs before and after the map-side merge operation For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Update to get the size of the key cluster , then go to step (3-4); (3-3) For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Perform statistical processing to obtain the size of the key cluster , then go to step (3-4); (3-4) Based on the size of each bond cluster obtained Get the key-value pair distribution number set of the key corresponding to the key cluster on all nodes Average value and standard deviation ; (3-5) The set of key-value pairs corresponding to each key cluster obtained in step (3-4) on each node Average value and standard deviation Get the distribution skewness index of the key cluster ; Specifically, step (5) includes the following sub-steps: (5-1) Obtain a set of bond clusters consisting of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a partition set consisting of multiple partitions from the Spark job obtained in step (3-1) ,in Indicates the total number of partitions, Indicates the partitions; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preference nodes ,in Indicates the preference nodes; (5-4) Set counter cnt1=1; (5-5) Determine whether cnt1 is greater than If yes, go to step (5-10), otherwise go to step (5-6); (5-6) Determine the cnt1th key cluster obtained in step (5-3) Is the total size less than or equal to the average partition size? If yes, go to step (5-7), otherwise go to step (5-8); Specifically, the total size of a key cluster group refers to the sum of the sizes of all key clusters in the key cluster group.

[0039] Specifically, the average partition size Equal to the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

[0040] (5-7) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) A partition that is not assigned a key cluster is obtained to obtain a partition that is assigned a key cluster, and then go to steps (5-9); (5-8) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) Multiple partitions that are not assigned key clusters (the allocation principle is that the partition size of each partition after the key cluster is assigned does not exceed the average partition size , and assign key clusters in the same group to as few partitions as possible, thereby minimizing the number of partitions used), to obtain multiple partitions to which key clusters are assigned, and then proceed to steps (5-9); (5-9) Set counter cnt1 = cnt1 + 1 and return to step (5-5); (5-10) Returns the partition set consisting of all partitions that have been assigned the key cluster as the preliminary partition set.

[0041] In more detail, steps (5-8) include the following sub-steps: (A1) Group the bond clusters All key clusters in are sorted in descending order according to the key cluster size to obtain the sorted key cluster group ,in Represents a key cluster group the total number of medium key clusters; (A2) Set counter cnt2=1; (A3) Determine whether cnt2 is greater than the key cluster group The total number of key clusters If yes, go to step (A9), otherwise go to step (A4); (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether there is a partition , meeting the following two conditions: 1) its current partition size is the largest in the partition set; 2) the partition After being assigned a key cluster, its current partition size does not exceed the average partition size If yes, go to step (A6), otherwise go to step (A7), where y represents the sequence number of the partition that meets the condition in the partition set; (A6) Cluster the bonds Assign to partition , to get an updated partition , then go to step (A8); (A7) Cluster the bonds Assigned to the partition set obtained in step (5-2) A partition that has not yet been assigned a key cluster to obtain a partition that has been assigned a key cluster Then go to step (A8); (A8) Set counter cnt2 = cnt2 + 1 and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions assigned key clusters.

[0042] Specifically, step (6) includes the following sub-steps: (6-1) Sort all key clusters with non-skewed distribution obtained in step (4) in descending order according to the size of the key cluster to obtain multiple sorted key clusters ,in represents the total number of bond clusters with non-skewed distribution obtained in step (4); (6-2) Set counter cnt3 = 1; (6-3) Determine whether cnt3 is greater than If yes, go to step (6-6), otherwise go to step (6-4); (6-4) Select the smallest partition from the partition set obtained in step (5-2) and add the cnt3th key cluster from the multiple key clusters sorted in step (6-1) to the partition set. Put it into the partition to obtain the updated partition, and then go to step (6-5); (6-5) Set counter cnt3 = cnt3 + 1 and return to step (6-3); (6-6) Return all updated partitions as the final partition set.

[0043] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for partitioning intermediate data based on data locality and load balancing awareness in a Spark framework, characterized in that: The following steps are involved: (1) Obtain the Spark jobs in the Spark cluster and the intermediate data stored in each node, and classify all the intermediate data according to their corresponding keys to obtain a key cluster set , where n is the total number of key categories, that is, the total number of key clusters in the key cluster set, represents the key cluster corresponding to the key of the i-th category, and i∈[1,n]; (2) Obtain each key cluster from the key cluster set obtained in step (1) Corresponding key The set of key-value pairs distributed across all nodes in the Spark cluster; (3) Each bond cluster obtained in step (2) Corresponding key The number of key-value pairs distributed on each node in the Spark cluster is collected to obtain the distribution skewness index of the key cluster. ; (4) The distribution skewness index of each bond cluster obtained in step (3) With preset parameters , get all bond clusters with skewed distribution and all bond clusters with non-skewed distribution; (5) Prioritize all key clusters with skewed distribution obtained in step (4) to obtain a preliminary partition set; (6) Based on the preliminary partition set obtained in step (5), load balancing is performed on all key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

2. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 1 is characterized in that: The preset parameters in step (4) satisfy , if the distribution skewness index of a key cluster Greater than parameter , then the key cluster is a key cluster with skew distribution. If the distribution skewness index of a key cluster is Less than or equal to the parameter , then the bond cluster is a bond cluster with non-skewed distribution.

3. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 1 or 2, characterized in that: Step (2) includes the following sub-steps: (2-1) Get the rack set consisting of all racks in the Spark cluster And all nodes in all racks, all nodes in all racks constitute the node set in Indicates the size of the rack set, Represents the size of the node set, and has ; (2-2) The node set obtained in step (2-1) Get each node Stored intermediate data , the intermediate data stored by all nodes constitute the intermediate data set ,in ∈[1, ], and there is: ； in Indicates the nodes Stored intermediate data The i-th key in Display key The corresponding j-th value, Indicates the nodes Stored intermediate data The middle button is , the value is Key-value pairs, For intermediate data The total number of keys included in , and there is ; (2-3) Get each key based on the intermediate data set obtained in step (2-2) At each node The corresponding key-value pair distribution number , this key The number of key-value pairs on all nodes constitutes the key-value pair distribution quantity set .

4. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to any one of claims 1 to 3, characterized in that: Step (3) includes the following sub-steps: (3-1) Determine whether there is a map-side combination operation triggered by a typical aggregation operator (such as reduceByKey, combineByKey, etc.) in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3); (3-2) The change ratio of the number of key-value pairs before and after the map-side merge operation For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Update to get the size of the key cluster , then go to step (3-4); (3-3) For each bond cluster obtained in step (2) Corresponding key A distribution set of key-value pairs across all nodes Perform statistical processing to obtain the size of the key cluster , then go to step (3-4); (3-4) Based on the size of each bond cluster obtained Get the key-value pair distribution number set of the key corresponding to the key cluster on all nodes Average value and standard deviation ; (3-5) The set of key-value pairs corresponding to each key cluster obtained in step (3-4) on each node Average value and standard deviation Get the distribution skewness index of the key cluster .

5. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 4 is characterized in that: Step (5) includes the following sub-steps: (5-1) Obtain a set of bond clusters consisting of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a partition set consisting of multiple partitions from the Spark job obtained in step (3-1) ,in Indicates the total number of partitions, Indicates the partitions; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preference nodes ,in Indicates the preference nodes; (5-4) Set counter cnt1=1; (5-5) Determine whether cnt1 is greater than If yes, go to step (5-10), otherwise go to step (5-6); (5-6) Determine the cnt1th key cluster obtained in step (5-3) Is the total size less than or equal to the average partition size? If yes, go to step (5-7), otherwise go to step (5-8); (5-7) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) A partition that is not assigned a key cluster is obtained to obtain a partition that is assigned a key cluster, and then go to steps (5-9); (5-8) Group the key clusters All key clusters in are assigned to the partition set obtained in step (5-2) Multiple partitions that are not assigned key clusters are obtained to obtain multiple partitions that are assigned key clusters, and then go to steps (5-9); (5-9) Set counter cnt1 = cnt1 + 1 and return to step (5-5); (5-10) Returns the partition set consisting of all partitions that have been assigned the key cluster as the preliminary partition set.

6. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 5, characterized in that: The total size of a key cluster group refers to the sum of the sizes of all key clusters in the key cluster group; Average partition size Equal to the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

7. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 6, characterized in that: Steps (5-8) include the following sub-steps: (A1) Group the bond clusters All key clusters in are sorted in descending order according to the key cluster size to obtain the sorted key cluster group ,in Represents a key cluster group the total number of medium key clusters; (A2) Set counter cnt2=1; (A3) Determine whether cnt2 is greater than the key cluster group The total number of key clusters If yes, go to step (A9), otherwise go to step (A4); (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether there is a partition , meeting the following two conditions: 1) its current partition size is the largest in the partition set; 2) the partition After being assigned a key cluster, its current partition size does not exceed the average partition size If yes, go to step (A6), otherwise go to step (A7), where y represents the sequence number of the partition that meets the condition in the partition set; (A6) Cluster the bonds Assign to partition , to get an updated partition , then go to step (A8); (A7) Cluster the bonds Assigned to the partition set obtained in step (5-2) A partition that has not yet been assigned a key cluster to obtain a partition that has been assigned a key cluster Then go to step (A8); (A8) Set counter cnt2 = cnt2 + 1 and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions assigned key clusters.

8. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 7, characterized in that: Step (6) includes the following sub-steps: (6-1) Sort all key clusters with non-skewed distribution obtained in step (4) in descending order according to the size of the key cluster to obtain multiple sorted key clusters ,in represents the total number of bond clusters with non-skewed distribution obtained in step (4); (6-2) Set counter cnt3 = 1; (6-3) Determine whether cnt3 is greater than If yes, go to step (6-6), otherwise go to step (6-4); (6-4) Select the smallest partition from the partition set obtained in step (5-2) and add the cnt3th key cluster from the multiple key clusters sorted in step (6-1) to the partition set. Put it into the partition to obtain the updated partition, and then go to step (6-5); (6-5) Set counter cnt3 = cnt3 + 1 and return to step (6-3); (6-6) Return all updated partitions as the final partition set.

9. An intermediate data partitioning system based on data locality and load balancing awareness in the Spark framework, characterized in that: include: The first module is used to obtain the Spark jobs in the Spark cluster and the intermediate data stored in each node, and classify all the intermediate data according to their corresponding keys to obtain a key cluster set. , where n is the total number of key categories, that is, the total number of key clusters in the key cluster set, represents the key cluster corresponding to the key of the i-th category, and i∈[1,n]; The second module is used to obtain each key cluster according to the key cluster set obtained by the first module. Corresponding key The set of key-value pairs distributed across all nodes in the Spark cluster; The third module is used for each key cluster obtained from the second module Corresponding key The number of key-value pairs distributed on each node in the Spark cluster is collected to obtain the distribution skewness index of the key cluster. ; The fourth module is used to calculate the distribution tilt index of each key cluster obtained from the third module. With preset parameters , get all bond clusters with skewed distribution and all bond clusters with non-skewed distribution; The fifth module is used to preferentially allocate all the key clusters with skewed distribution obtained by the fourth module to obtain a preliminary partition set; The sixth module is used to perform load balancing on all key clusters with non-skewed distribution obtained in the fourth module based on the preliminary partition set obtained in the fifth module to obtain a final partition set.

Images