Data locality and load balancing aware intermediate data partitioning method and system in spark framework

Abstract

This invention discloses an intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework. First, for the large amount of intermediate key-value pair data generated during the Shuffle stage, the distribution of each key cluster across the nodes of the Spark cluster is statistically analyzed, constructing a distribution skew model that quantifies the degree of data imbalance between nodes, thereby obtaining more realistic and comprehensive data distribution information. Then, all key clusters are classified using the distribution skew index, dividing them into skewed and non-skewed distributions to achieve differentiated allocation strategy design. Next, for key clusters with skewed distributions, the concept of a preferred node is introduced, prioritizing the allocation of multiple key clusters densely distributed on the same preferred node to the same partition, thereby enhancing the local concentration of data and optimizing the data locality of the reduce task. Finally, based on the preliminary partitioning results, a load balancing allocation strategy is applied to non-skewed key clusters.

Description

Technical Field

[0001] This invention belongs to the field of big data and distributed parallel computing technology, and more specifically, relates to an intermediate data partitioning method and system based on data locality and load balancing awareness in the Spark framework. Background Technology

[0002] Spark is a big data processing framework based on an in-memory computing model, widely used in iterative computation, interactive queries, and stream processing. During Spark execution, raw input data and intermediate processing results are typically organized as partitions. Each partition is processed by a single task, and multiple tasks execute in parallel across multiple nodes in the cluster. A typical Spark job usually consists of multiple stages: a map stage (the first stage) and a reduce stage (all subsequent stages). The map stage generates intermediate results from the input data, while the reduce stage processes the intermediate key-value pairs generated by the previous stage, completing the final computation. Wide dependencies exist between map and reduce stages, or between reduce stages themselves. Spark needs to perform a shuffle operation, meaning that a reduce task processing a partition needs to pull its portion of intermediate data from various nodes to ensure that each reduce task receives all the data it needs. Therefore, when the amount of intermediate data distributed across different partitions is severely uneven, it leads to significant differences in the reduce task load, a phenomenon known as data skew. This can cause abnormally long execution times for some tasks, lowering the overall job completion efficiency. Furthermore, the intermediate data partitioning method determines the distribution of key-value pairs within partitions, which also affects the effectiveness of reduce tasks in utilizing data locality for optimization. This is because reduce tasks are prioritized for scheduling to the node storing the largest amount of data in their processed partition, minimizing cross-node data transfer and reducing network communication overhead. Therefore, a reasonable data partitioning method is crucial in the Spark framework. It not only affects task parallelism and data locality but also effectively alleviates problems such as uneven computational load and communication bottlenecks, directly determining the overall job execution efficiency and resource utilization.

[0003] Currently, the Spark framework primarily supports two data partitioning methods by default. The first method is hash-based partitioning, which distributes key-value pairs evenly across different partitions by hashing the keys and then taking the modulo of the number of partitions. This strategy is simple to implement and suitable for scenarios where the key-value distribution is relatively balanced. The second method is range-based partitioning, which uses sampling to obtain the global sorting information of the keys and divides the keys into several continuous intervals, thereby achieving ordering between partitions. This method is often used for operations such as sorting and window aggregation.

[0004] However, the above methods all have some significant drawbacks: First, hash-based data partitioning assumes a uniform key distribution, but data skew often occurs in real-world data, leading to overloaded partitions and impacting overall job performance. Second, range-based data partitioning can alleviate data skew to some extent, but its partition boundaries depend on the representativeness of the sampled data, making it susceptible to sample bias and resulting in uneven data volume in some partitions. Third, neither of these data partitioning methods considers the impact on data locality of the reduce task, limiting the effectiveness of network communication optimization. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a data locality- and load-balance-aware intermediate data partitioning method and system within the Spark framework. Its purpose is to resolve the technical problems of existing hash-based data partitioning methods, which often suffer from data skew in real-world data, leading to excessive load on some partitions and impacting overall job performance; the technical problems of existing range-based data partitioning methods, where partition boundaries depend on the representativeness of sampled data and are easily affected by sample bias, resulting in uneven data volume in some partitions; and the lack of data locality awareness for reduce tasks in both of these data partitioning methods, leading to high communication overhead due to the need to transmit large amounts of intermediate data over the network during the shuffle process.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for intermediate data partitioning based on data locality and load balancing awareness in the Spark framework is provided, comprising the following steps: (1) Obtain the Spark jobs and intermediate data stored on each node in the Spark cluster, and classify all intermediate data according to their corresponding keys to obtain a key cluster set. Where n is the total number of key categories, which is the total number of key clusters in the key cluster set. Let i represent the key cluster corresponding to the key of the i-th category, where i ∈ [1, n]. (2) Obtain each key cluster from the key cluster set obtained in step (1). Corresponding key The set of key-value pair distributions across all nodes in a Spark cluster.

[0007] (3) Each key cluster obtained according to step (2) Corresponding key In a Spark cluster, the distribution skewness index is obtained by analyzing the number of key-value pairs distributed across each node. .

[0008] (4) The distribution tilt index of each bond cluster obtained in step (3) With preset parameters Obtain all bond clusters with skewed distributions and all bond clusters with non-skewed distributions; (5) Prioritize the allocation of all key clusters with skewed distributions obtained in step (4) to obtain a preliminary partition set.

[0009] (6) Based on the preliminary partition set obtained in step (5), load balance is applied to all the key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

[0010] Preferably, the preset parameters in step (4) satisfy... If the distribution skewness index of a certain bond cluster Greater than the parameter If the distribution skewness index of a certain bond cluster is... Less than or equal to parameter If so, then the bond cluster is a bond cluster with a non-tilted distribution.

[0011] Preferably, step (2) includes the following sub-steps: (2-1) Obtain the set of racks composed of all the architectures in the Spark cluster. And all nodes in all racks, which together constitute the node set. in Indicates the size of the rack assembly. This represents the size of the node set, and has... (2-2) The set of nodes obtained in step (2-1) Get each node Storing intermediate data The intermediate data stored by all nodes constitutes the intermediate data set. ,in ∈[1, ], and there is: ; in Indicates the first Nodes Storing intermediate data The i-th key in Indicator key The corresponding j-th value, Indicates the first Nodes Storing intermediate data middle key is Value key-value pairs, intermediate data The total number of keys included, and have ; (2-3) Obtain each key based on the intermediate data set obtained in step (2-2). At each node The corresponding key-value pair distribution number The key The number of key-value pairs across all nodes constitutes the set of key-value pair distributions. .

[0012] Preferably, step (3) includes the following sub-steps: (3-1) Determine whether there is a map-side combination operation triggered by typical aggregation operators (such as reduceByKey, combineByKey, etc.) in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3). (3-2) The change in the number of key-value pairs before and after the map-side merge operation For each key cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Update to obtain the size of the bond cluster. Then proceed to steps (3-4); (3-3) For each key cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Perform statistical processing to obtain the size of the bond cluster. Then proceed to steps (3-4); (3-4) Based on the size of each obtained bond cluster Get the set of key-value pair distributions of the key corresponding to this key cluster across all nodes. average and standard deviation ; (3-5) The set of key-value pair distributions of each key cluster corresponding to each node obtained in step (3-4). average and standard deviation Obtain the distribution tilt index of the bond cluster. ; Preferably, step (5) includes the following sub-steps: (5-1) Obtain the set of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a set of partitions consisting of multiple partitions from the Spark job obtained in step (3-1). ,in Indicates the total number of partitions. Indicates the first One partition; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preferred nodes ,in Indicates the first One preference node; (5-4) Set the counter cnt1=1; (5-5) Determine if cnt1 is greater than If yes, proceed to step (5-10); otherwise, proceed to step (5-6). (5-6) Determine the cnt1th bond cluster group obtained in step (5-3). Is the total size less than or equal to the average partition size? If yes, proceed to step (5-7); otherwise, proceed to step (5-8). (5-7) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Take a partition that has not been assigned a key cluster to obtain a partition that has been assigned a key cluster, and then proceed to steps (5-9). (5-8) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Multiple partitions without assigned key clusters are processed to obtain multiple partitions with assigned key clusters, and then proceed to steps (5-9). (5-9) Set the counter cnt1 = cnt1 + 1, and return to step (5-5); (5-10) Return the set of partitions consisting of all partitions that have been assigned key clusters as the initial set of partitions.

[0013] Preferably, the total size of the bond cluster group refers to the sum of the sizes of all bond clusters in the bond cluster group.

[0014] Average partition size Equals the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

[0015] Preferably, step (5-8) includes the following sub-steps: (A1) group the bond clusters All key clusters are sorted in descending order of key cluster size to obtain the sorted key cluster group. ,in Represents a bond cluster group The total number of intermediate bond clusters; (A2) Set the counter cnt2=1; (A3) Determine if cnt2 is greater than the bond cluster set. Total number of intermediate bond clusters If yes, proceed to step (A9); otherwise, proceed to step (A4). (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether a partition exists. It meets the following two conditions: 1) Its current partition size is the largest in the partition set; 2) This partition After a key cluster is assigned, its current partition size does not exceed the average partition size. If yes, proceed to step (A6); otherwise, proceed to step (A7), where y represents the index of the partition that meets the condition in the partition set. (A6) Bond cluster Assigned to partition To obtain an updated partition Then proceed to step (A8); (A7) Bond cluster Assign to the partition set obtained in step (5-2) To obtain a partition that has not yet been assigned a key cluster, select a partition that has been assigned a key cluster. The partition is then proceeded to step (A8); (A8) Set the counter cnt2 = cnt2 + 1, and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions that have been assigned key clusters.

[0016] Preferably, step (6) includes the following sub-steps: (6-1) Sort all the non-skewed bond clusters obtained in step (4) in descending order by the size of the bond clusters to obtain multiple sorted bond clusters. ,in This represents the total number of all bond clusters with non-tilted distributions obtained in step (4); (6-2) Set the counter cnt3=1; (6-3) Determine if cnt3 is greater than If yes, proceed to step (6-6); otherwise, proceed to step (6-4). (6-4) Select the smallest partition from the partition set obtained in step (5-2), and then select the cnt3rd key cluster from the multiple key clusters sorted in step (6-1). Place it into this partition to obtain the updated partition, and then proceed to step (6-5). (6-5) Set the counter cnt3 = cnt3 + 1, and return to step (6-3); (6-6) Return all updated partitions as the final set of partitions.

[0017] According to another aspect of the present invention, a data locality- and load balancing-aware intermediate data partitioning system in the Spark framework is provided, comprising: The first module is used to obtain Spark jobs and intermediate data stored on each node in the Spark cluster, and to categorize all intermediate data according to their corresponding keys to obtain a set of key clusters. Where n is the total number of key categories, which is the total number of key clusters in the key cluster set. Let i represent the key cluster corresponding to the key of the i-th category, where i ∈ [1, n]. The second module is used to obtain each key cluster from the key cluster set obtained in the first module. Corresponding key The set of key-value pair distributions across all nodes in a Spark cluster.

[0018] The third module is used to process each key cluster obtained from the second module. Corresponding key In a Spark cluster, the distribution skewness index is obtained by analyzing the number of key-value pairs distributed across each node. .

[0019] The fourth module is used to determine the distribution tilt index of each bond cluster obtained from the third module. With preset parameters Obtain all bond clusters with skewed distributions and all bond clusters with non-skewed distributions; The fifth module is used to preferentially allocate all the key clusters with skewed distributions obtained from the fourth module to obtain a preliminary set of partitions.

[0020] The sixth module is used to perform load balancing on all key clusters with non-skewed distributions obtained from the fourth module based on the preliminary partition set obtained from the fifth module, so as to obtain the final partition set.

[0021] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) Since the present invention adopts steps (1) to (4), it can distinguish the data distribution patterns of different key clusters between nodes by introducing a key cluster distribution tilt modeling method, thereby identifying key clusters with obvious tilt characteristics and realizing explicit modeling of the non-uniformity of intermediate data between nodes; therefore, it can effectively solve the technical problem that the existing Hash-based data partitioning method assumes uniform key distribution and is difficult to cope with the uneven load caused by data tilt. (2) Because the present invention adopts step (4), it makes full use of the global real distribution information when classifying key clusters, rather than relying on sampled data for division, and calculates and classifies the tilt based on the actual distribution; therefore, it can effectively solve the technical problems of existing Range-based partitioning methods that are highly dependent on sampling results and easily affected by sample bias, thereby improving the accuracy and robustness of partition boundary division. (3) In step (5), the present invention introduces a priority allocation strategy. For key clusters with skewed distribution, multiple key clusters distributed on the same preferred node are allocated to the same partition as much as possible to enhance the data concentration of the partition on the node, thereby maximizing the data locality of the reduce task, reducing cross-node data transmission, and effectively reducing network communication overhead and scheduling delay during the Shuffle process. Therefore, it can effectively solve the problem of high communication overhead caused by the lack of data locality awareness of the reduce task in existing Hash-based data partitioning methods and Range-based partitioning methods, which requires a large amount of intermediate data to be transmitted across nodes during the Shuffle process.

[0022] (4) In step (6), the present invention proposes a load balancing allocation strategy for non-skewed distribution key clusters, which dynamically adjusts the allocation of remaining key clusters in combination with the existing initial partitioning results to achieve overall balance at the partitioning granularity, thereby improving the execution efficiency of parallel tasks and ensuring the stability and scalability of the job. Attached Figure Description

[0023] Figure 1 This is a flowchart of the intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework of this invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The basic idea of ​​this invention is to jointly optimize the partitioning effect of Spark intermediate data from four aspects. First, for the large amount of intermediate key-value pair data generated in the Shuffle stage, the distribution of each key cluster on each node of the Spark cluster is statistically analyzed, and a distribution skew model that can quantify the degree of data imbalance between nodes is constructed, thereby obtaining more realistic and comprehensive data distribution information. Second, all key clusters are classified using the distribution skew index, dividing them into skewed and non-skewed distributions to achieve differentiated allocation strategy design. Then, for key clusters with skewed distributions, the concept of a preferred node is introduced, prioritizing the allocation of multiple key clusters densely distributed on the same preferred node to the same partition, thereby enhancing the local concentration of data and achieving optimal data locality for reduce tasks. Finally, based on the preliminary partitioning results, a load balancing allocation strategy is adopted for non-skewed key clusters to balance the total amount of data in each partition, improve parallel execution efficiency, and ultimately significantly reduce network communication overhead and performance bottlenecks caused by data skew while ensuring job execution stability.

[0026] The following is an explanation and description of some technical terms used in this invention: A key cluster is a collection of all key-value pairs that share the same key.

[0027] Preference node: For a given key cluster, its preference node is defined as the node that stores the largest number of key / value pairs corresponding to that key cluster among all nodes.

[0028] like Figure 1 As shown, this invention provides a method for intermediate data partitioning based on data locality and load balancing awareness in the Spark framework, including the following steps: (1) Obtain the Spark jobs and intermediate data (key-value pairs) stored on each node in the Spark cluster, and classify all intermediate data according to their corresponding keys to obtain a key cluster set. Where n is the total number of key categories, which is the total number of key clusters in the key cluster set. Let i represent the key cluster corresponding to the key of the i-th category, where i ∈ [1, n].

[0029] (2) Obtain each key cluster from the key cluster set obtained in step (1). Corresponding key The set of key-value pair distributions across all nodes in a Spark cluster.

[0030] (3) Each key cluster obtained according to step (2) Corresponding key In a Spark cluster, the distribution skewness index is obtained by analyzing the number of key-value pairs distributed across each node. .

[0031] (4) The distribution tilt index of each bond cluster obtained in step (3) With preset parameters Obtain all bond clusters with skewed distributions and all bond clusters with non-skewed distributions; Specifically, preset parameters If the distribution skewness index of a certain bond cluster Greater than the parameter If the distribution skewness index of a certain bond cluster is... Less than or equal to parameter If so, then the bond cluster is a bond cluster with a non-tilted distribution.

[0032] The advantage of step (4) is that it realizes the classification of key clusters, making the entire partitioning process more adaptive and dynamically adjustable. By decoupling the processing of skewed and non-skewed data, the versatility and robustness of the system under multiple scenarios and data characteristics are improved.

[0033] (5) Prioritize the allocation of all key clusters with skewed distributions obtained in step (4) to obtain a preliminary partition set.

[0034] The advantage of this step (5) is that it adopts a priority allocation strategy for key clusters with skewed distribution, which not only ensures data locality but also avoids excessive load on a single partition, thus achieving an effective balance between partition load balancing and data skew mitigation.

[0035] (6) Based on the preliminary partition set obtained in step (5), load balance is applied to all the key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

[0036] The advantage of steps (1) to (6) above is that it systematically combines data locality and load balancing awareness, and achieves an efficient intermediate data partitioning and allocation strategy by differentiating between skewed and non-skewed key clusters, effectively alleviating the data skew problem and improving the execution efficiency and resource utilization of Spark jobs.

[0037] Specifically, step (2) includes the following sub-steps: (2-1) Obtain the set of racks composed of all the architectures in the Spark cluster. And all nodes in all racks, which together constitute the node set. in Indicates the size of the rack assembly. This represents the size of the node set, and has... (2-2) The set of nodes obtained in step (2-1) Get each node Storing intermediate data The intermediate data stored by all nodes constitutes the intermediate data set. ,in ∈[1, ], and there is: ; in Indicates the first Nodes Storing intermediate data The i-th key in Indicator key The corresponding j-th value, Indicates the first Nodes Storing intermediate data middle key is Value key-value pairs, intermediate data The total number of keys included, and have ; (2-3) Obtain each key based on the intermediate data set obtained in step (2-2). At each node The corresponding key-value pair distribution number The key The number of key-value pairs across all nodes constitutes the set of key-value pair distributions. .

[0038] Specifically, step (3) includes the following sub-steps: (3-1) Determine whether there is a map-side combination operation triggered by typical aggregation operators (such as reduceByKey, combineByKey, etc.) in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3). (3-2) The change in the number of key-value pairs before and after the map-side merge operation For each key cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Update to obtain the size of the bond cluster. Then proceed to steps (3-4); (3-3) For each key cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Perform statistical processing to obtain the size of the bond cluster. Then proceed to steps (3-4); (3-4) Based on the size of each obtained bond cluster Get the set of key-value pair distributions of the key corresponding to this key cluster across all nodes. average and standard deviation ; (3-5) The set of key-value pair distributions of each key cluster corresponding to each node obtained in step (3-4). average and standard deviation Obtain the distribution tilt index of the bond cluster. ; Specifically, step (5) includes the following sub-steps: (5-1) Obtain the set of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a set of partitions consisting of multiple partitions from the Spark job obtained in step (3-1). ,in Indicates the total number of partitions. Indicates the first One partition; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preferred nodes ,in Indicates the first One preference node; (5-4) Set the counter cnt1=1; (5-5) Determine if cnt1 is greater than If yes, proceed to step (5-10); otherwise, proceed to step (5-6). (5-6) Determine the cnt1th bond cluster group obtained in step (5-3). Is the total size less than or equal to the average partition size? If yes, proceed to step (5-7); otherwise, proceed to step (5-8). Specifically, the total size of a bond cluster group refers to the sum of the sizes of all bond clusters in that group.

[0039] Specifically, average partition size Equals the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

[0040] (5-7) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Take a partition that has not been assigned a key cluster to obtain a partition that has been assigned a key cluster, and then proceed to steps (5-9). (5-8) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Multiple partitions that have not been assigned key clusters (the allocation principle is that the size of each partition after being assigned a key cluster does not exceed the average partition size). And allocate the key clusters in the same group to as few partitions as possible, thereby minimizing the number of partitions used), to obtain multiple partitions with key clusters allocated, and then proceed to step (5-9). (5-9) Set the counter cnt1 = cnt1 + 1, and return to step (5-5); (5-10) Return the set of partitions consisting of all partitions that have been assigned key clusters as the initial set of partitions.

[0041] Furthermore, steps (5-8) include the following sub-steps: (A1) group the bond clusters All key clusters are sorted in descending order of key cluster size to obtain the sorted key cluster group. ,in Represents a bond cluster group The total number of intermediate bond clusters; (A2) Set the counter cnt2=1; (A3) Determine if cnt2 is greater than the bond cluster set. Total number of intermediate bond clusters If yes, proceed to step (A9); otherwise, proceed to step (A4). (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether a partition exists. It meets the following two conditions: 1) Its current partition size is the largest in the partition set; 2) This partition After a key cluster is assigned, its current partition size does not exceed the average partition size. If yes, proceed to step (A6); otherwise, proceed to step (A7), where y represents the index of the partition that meets the condition in the partition set. (A6) Bond cluster Assigned to partition To obtain an updated partition Then proceed to step (A8); (A7) Bond cluster Assign to the partition set obtained in step (5-2) To obtain a partition that has not yet been assigned a key cluster, select a partition that has been assigned a key cluster. The partition is then proceeded to step (A8); (A8) Set the counter cnt2 = cnt2 + 1, and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions that have been assigned key clusters.

[0042] Specifically, step (6) includes the following sub-steps: (6-1) Sort all the non-skewed bond clusters obtained in step (4) in descending order by the size of the bond clusters to obtain multiple sorted bond clusters. ,in This represents the total number of all bond clusters with non-tilted distributions obtained in step (4); (6-2) Set the counter cnt3=1; (6-3) Determine if cnt3 is greater than If yes, proceed to step (6-6); otherwise, proceed to step (6-4). (6-4) Select the smallest partition from the partition set obtained in step (5-2), and then select the cnt3rd key cluster from the multiple key clusters sorted in step (6-1). Place it into this partition to obtain the updated partition, and then proceed to step (6-5). (6-5) Set the counter cnt3 = cnt3 + 1, and return to step (6-3); (6-6) Return all updated partitions as the final set of partitions.

[0043] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intermediate data partitioning based on data locality and load balancing awareness in Spark framework, characterized in that, Includes the following steps: (1) Obtain the Spark jobs and intermediate data stored on each node in the Spark cluster, and classify all intermediate data according to their corresponding keys to obtain a key cluster set. Where n is the total number of key categories, which is the total number of key clusters in the key cluster set. Let i represent the key cluster corresponding to the key of the i-th category, where i ∈ [1, n]. (2) Obtain each key cluster from the key cluster set obtained in step (1). Corresponding key The set of key-value pair distributions across all nodes in a Spark cluster; (3) Each key cluster obtained according to step (2) Corresponding key In a Spark cluster, the distribution skewness index is obtained by analyzing the number of key-value pairs distributed across each node. ; (4) The distribution tilt index of each bond cluster obtained in step (3) With preset parameters Obtain all bond clusters with skewed distributions and all bond clusters with non-skewed distributions; (5) Prioritize the allocation of all key clusters with skewed distributions obtained in step (4) to obtain a preliminary partition set; (6) Based on the preliminary partition set obtained in step (5), load balance is applied to all the key clusters with non-skewed distribution obtained in step (4) to obtain the final partition set.

2. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 1, characterized in that, The preset parameters in step (4) satisfy If the distribution skewness index of a certain bond cluster Greater than the parameter If the distribution skewness index of a certain bond cluster is... Less than or equal to parameter If so, then the bond cluster is a bond cluster with a non-tilted distribution.

3. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 1 or 2, characterized in that, Step (2) includes the following sub-steps: (2-1) Obtain the set of racks composed of all the architectures in the Spark cluster. And all nodes in all racks, which together constitute the node set. in Indicates the size of the rack assembly. This represents the size of the node set, and has... ; (2-2) The set of nodes obtained in step (2-1) Get each node Storing intermediate data The intermediate data stored by all nodes constitutes the intermediate data set. ,in ∈[1, ], and there is: ;in Indicates the first Nodes Storing intermediate data The i-th key in Indicator key The corresponding j-th value, Indicates the first Nodes Storing intermediate data middle key is Value key-value pairs, intermediate data The total number of keys included, and have ; (2-3) Obtain each key based on the intermediate data set obtained in step (2-2). At each node The corresponding key-value pair distribution number The key The number of key-value pairs across all nodes constitutes the set of key-value pair distributions. .

4. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 3, characterized in that, Step (3) includes the following sub-steps: (3-1) Determine whether there is a map-side merge operation triggered by a typical aggregation operator in the Spark job. If so, proceed to step (3-2); otherwise, proceed to step (3-3). (3-2) The change in the number of key-value pairs before and after the map-side merge operation For each key cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Update to obtain the size of the bond cluster. Then proceed to steps (3-4); (3-3) For each bond cluster obtained in step (2) Corresponding key Key-value pair distribution set across all nodes Perform statistical processing to obtain the size of the bond cluster. Then proceed to steps (3-4). (3-4) Based on the size of each obtained bond cluster Get the set of key-value pair distributions of the key corresponding to this key cluster across all nodes. average and standard deviation ; (3-5) The set of key-value pair distributions of each key cluster corresponding to each node obtained in step (3-4). average and standard deviation Obtain the distribution tilt index of the bond cluster. .

5. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 4, characterized in that, Step (5) includes the following sub-steps: (5-1) Obtain the set of all bond clusters with skewed distribution obtained in step (4); (5-2) Obtain a set of partitions consisting of multiple partitions from the Spark job obtained in step (3-1). ,in Indicates the total number of partitions. Indicates the first One partition; (5-3) Divide the set of bond clusters with skewed distribution obtained in step (5-1) to obtain Key clusters with preferred nodes ,in Indicates the first One preference node; (5-4) Set the counter cnt1=1; (5-5) determine whether cntl is greater than If so, go to step (5-10), otherwise go to step (5-6); (5-6) Determine the cnt1th bond cluster group obtained in step (5-3). Is the total size less than or equal to the average partition size? If yes, proceed to step (5-7); otherwise, proceed to step (5-8). (5-7) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Take a partition that has not been assigned a key cluster to obtain a partition that has been assigned a key cluster, and then proceed to steps (5-9). (5-8) group the bond clusters All key clusters are assigned to the partition set obtained in step (5-2). Multiple partitions without assigned key clusters are processed to obtain multiple partitions with assigned key clusters, and then proceed to steps (5-9). (5-9) Set the counter cnt1 = cnt1 + 1, and return to step (5-5); (5-10) Return the set of partitions consisting of all partitions that have been assigned key clusters as the initial set of partitions.

6. The intermediate data partitioning method based on data locality and load balancing awareness in the Spark framework according to claim 5, characterized in that, The total size of a bond cluster group refers to the sum of the sizes of all bond clusters in that bond cluster group; Average partition size Equals the sum of the sizes of all key clusters in the Spark job / the total number of partitions .

7. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 6, characterized in that, Steps (5-8) include the following sub-steps: (A1) sort all key-clusters in the key-cluster group in descending order of key-cluster size to obtain a sorted key-cluster group wherein denotes the total number of key-clusters in the key-cluster group . (A2) Set the counter cnt2=1; (A3) judge whether cnt2 is greater than the number of key cluster groups the total number of key clusters if yes, go to step (A9), otherwise go to step (A4); (A4) Obtain the current partition size of each partition in the partition set obtained in step (5-2); (A5) Based on the current partition size of each partition in the partition set obtained in step (A4), determine whether a partition exists. It meets the following two conditions: 1) Its current partition size is the largest in the partition set; 2) This partition After a key cluster is assigned, its current partition size does not exceed the average partition size. If yes, proceed to step (A6); otherwise, proceed to step (A7), where y represents the index of the partition that meets the condition in the partition set. (A6) the key cluster is assigned to a partition to obtain an updated partition and go to step (A8).​ (A7) Bond cluster Assign to the partition set obtained in step (5-2) To obtain a partition that has not yet been assigned a key cluster, select a partition that has been assigned a key cluster. The partition is then proceeded to step (A8); (A8) Set the counter cnt2 = cnt2 + 1, and return to step (A3); (A9) Returns a set of partitions consisting of multiple partitions that have been assigned key clusters.

8. The method for data locality and load balancing awareness based intermediate data partitioning in Spark framework according to claim 7, characterized in that, Step (6) includes the following sub-steps: (6-1) Sort all the non-skewed bond clusters obtained in step (4) in descending order by the size of the bond clusters to obtain multiple sorted bond clusters. ,in This represents the total number of all bond clusters with non-tilted distributions obtained in step (4); (6-2) Set the counter cnt3=1; (6-3) Determine if cnt3 is greater than If yes, proceed to step (6-6); otherwise, proceed to step (6-4). (6-4) Select the smallest partition from the partition set obtained in step (5-2), and then select the cnt3rd key cluster from the multiple key clusters sorted in step (6-1). Place it into the partition to obtain the updated partition, and then proceed to step (6-5). (6-5) Set the counter cnt3 = cnt3 + 1, and return to step (6-3); (6-6) Return all updated partitions as the final set of partitions.

9. A system for data locality and load balancing aware intermediate data partitioning in Spark framework, characterized in that, include: The first module is used to obtain Spark jobs and intermediate data stored on each node in the Spark cluster, and to categorize all intermediate data according to their corresponding keys to obtain a set of key clusters. Where n is the total number of key categories, which is the total number of key clusters in the key cluster set. Let i represent the key cluster corresponding to the key of the i-th category, where i ∈ [1, n]. A second module is configured to obtain each key cluster from the key cluster set obtained by the first module Corresponding key A set of key-value pair distribution quantities on all nodes in the Spark cluster The third module is used to process each key cluster obtained from the second module. Corresponding key In a Spark cluster, the distribution skewness index is obtained by analyzing the number of key-value pairs distributed across each node. ; The fourth module is used to determine the distribution tilt index of each bond cluster obtained from the third module. With preset parameters Obtain all bond clusters with skewed distributions and all bond clusters with non-skewed distributions; The fifth module is used to preferentially allocate all the key clusters with skewed distributions obtained in the fourth module to obtain a preliminary set of partitions; The sixth module is used to perform load balancing on all key clusters with non-skewed distributions obtained from the fourth module based on the preliminary partition set obtained from the fifth module, so as to obtain the final partition set.

Images