Data skew optimization method based on Spark
By identifying and processing skewed data in advance on the Map task side, the long-tail task problem caused by data skewed in the Spark computing framework is solved, and the computing performance and resource utilization efficiency are improved. It is suitable for all tasks containing shuffle processes.
Patent Information
- Application Number
- CN202510544595.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-05
AI Technical Summary
The existing Spark computing framework has data tilt problems in the data shuffle process, resulting in long-tail tasks and affecting the efficiency of analysis tasks. The existing optimization solution is only applicable to the two-table join calculation, and the resource consumption increases, and is not applicable to the calculation tasks of other shuffle processes.
On the Map task side, the tilt partition is identified in advance and the tilt data is processed. The tilt Reduce partition and shuffle key are identified through coarse and fine-grained size. The tilt data is discarded to avoid the tilt data being shuffled to the Reduce end, which is suitable for all calculation tasks containing shuffle processes.
It effectively avoids long-tail tasks, reduces resource consumption, improves the computing performance and efficiency of shuffle-like Spark tasks, and is suitable for computing tasks of various shuffle processes.
Smart Images
Figure CN120429367A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of big data and database, and in particular relates to a Spark-based data tilt optimization method. Background Art
[0002] Spark is a highly efficient computing framework, particularly well-suited for offline analysis of massive amounts of data. Most Spark analysis tasks, such as association calculations and grouped aggregate statistics, rely on data shuffling. Shuffling involves "shuffling" data according to specific partitioning rules, consuming significant amounts of disk and network I / O, thus becoming a bottleneck for massive data analysis performance. Data skew during the shuffle process can lead to "long-tail" tasks, prolonging the overall computational time and severely impacting analysis efficiency.
[0003] The open-source Spark community provides a solution to the data skew problem that occurs when joining two tables. The corresponding optimization rule is OptimizeSkewedJoin, which is one of the AQE optimization strategies. The main optimization idea is to split the skewed partitions in the two tables participating in the join into smaller partitions and copy the small partitions of one table to the matching partitions of the other table. This strategy can avoid the "long tail" tasks caused by data skew, reduce the time consumption of the shuffle process, and improve the computational efficiency of the join calculation.
[0004] Currently, open source Spark's optimization of skewed data also has shortcomings: Increased resource (task) consumption: Splitting skewed partitions increases resource demand, meaning the number of required tasks increases. If the matching partitions of both the left and right tables are skewed, and the number of subpartitions from the skewed partitions is M and N, respectively, the number of Reduce tasks required after splitting the subpartitions is the Cartesian product (M*N) of the number of subpartitions from the left and right tables.
[0005] This is not a universal data skew optimization solution: This solution is only effective for two-table join scenarios and is not applicable to other computing tasks that include shuffle processes. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide a Spark-based data skew optimization method to address the deficiencies of the background technology. The purpose of the present invention is to identify skewed partitions in advance on the Map task side and process skewed data according to user needs to avoid the skewed data being shuffled to the Reduce side, which causes the skewed Reduce to take a long time to pull the shuffle data, resulting in long-tail tasks and affecting the operating efficiency of the entire analysis task. Moreover, this data skew optimization method is not only applicable to the two-table Join calculation type, but also to all calculation tasks that include the Shuffle process, and can improve the computing performance of Shuffle-type Spark tasks.
[0007] The present invention adopts the following technical solutions to solve the above technical problems: A Spark-based data skew optimization method adds a skewed data identification and processing module to the Map task. It is divided into three stages: In the first stage, coarse-grained identification of skewed Reduce partitions is used: When writing the first several batches of data on the Map side, the Reduce partitions to which the skewed shuffle keys belong are identified through coarse-grained identification. Phase 2: Identify hot shuffle keys in fine-grained fashion; find the shuffle keys that cause the skew in the Reduce partitions to which they belong. Phase 3: Processing skewed data: Process all skewed shuffle keys identified in Phase 2.
[0008] As a further preferred solution of the Spark-based data skew optimization method of the present invention, in the first stage, coarse-grained identification of skewed Reduce partitions specifically includes the following steps: Step A1: Divide the data into batches. Each batch contains a fixed number of data items, which is defined as batch_size. In step A2, when ShuffleWriter receives the first M batches of data and writes [K, V] data to the buffer, it first counts the number of entries in each Reduce partition. If the number of data entries in some Reduce partitions in the first M batches exceeds a certain threshold and is greater than a certain multiple of the median number of data entries in all partitions, it is considered a skewed partition.
[0009] As a further preferred solution of the Spark-based data tilt optimization method of the present invention, in the second stage, fine-grained identification of hotspot shuffle keys specifically includes the following steps: Step B1: Based on the Reduce partition where the skewed data is located, the shuffle key causing the skew is further found in the skewed Reduce partition. In step B2, skewed shuffle keys are identified through the Map output data from batches M+1 to N. Specifically, before writing data to the buffer, the total number of data items for each shuffle key is counted. After the Nth batch of data is sent, shuffle keys whose total number of data items is greater than a certain threshold and greater than a certain multiple of the median number of data items of all shuffle keys are identified as skewed shuffle keys.
[0010] As a further preferred solution of the Spark-based data tilt optimization method of the present invention, in the third stage, processing the tilted data specifically includes the following steps: In step C1, based on all skewed shuffle keys identified in the second phase, skewed data processing begins. In actual application scenarios, most skewed data is invalid data. Therefore, for identified skewed shuffle keys, the skewed shuffle keys are discarded by default according to the skewed data discard ratio and no longer participate in the shuffle process. Step C2: When the client sends SQL, you can specify the discard ratio of the tilted shuffle key through a hint. If the client specifies the discard ratio, the tilted shuffle key is discarded according to the discard ratio specified by the client. In step C3, the tilted shuffle key data is sampled and recorded, and the total number of all tilted shuffle keys in each Map task is counted and recorded in the statistical table.
[0011] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: The present invention provides a Spark-based data skew optimization method. The purpose of the present invention is to identify skewed partitions in advance on the Map task side and process the skewed data according to user needs, so as to prevent the skewed data from being shuffled to the Reduce side, which causes the skewed Reduce to take a long time to pull the shuffle data, resulting in long-tail tasks and affecting the operating efficiency of the entire analysis task. In addition, this data skew optimization method is not only applicable to the two-table Join calculation type, but also to all calculation tasks involving the Shuffle process, and can improve the computing performance of Shuffle-type Spark tasks. 2. Spark's native skewed partitions are calculated based on the statistical information of the shuffle data after the shuffle data is written on the Map side. At this time, all the shuffle data has been written to the RSS Worker, and the shuffle process still consumes a lot of disk and network IO. The problem of long shuffle time cannot be effectively solved. The present invention proposes a method of first using a small amount of data to identify skewed partitions in the first few data processing batches at the beginning of the Map task's calculation process, and then further using a small amount of data to identify skewed shuffle keys. The method supports users to specify how to handle skewed shuffle data, improves the flexibility of using Spark for offline analysis in actual application scenarios, reduces the participation of invalid data in the shuffle process, saves resource consumption in the shuffle process, reduces the time consumption of the shuffle process, and thus improves the execution efficiency of shuffle-type Spark computing tasks. 3. This invention is a general solution that is applicable to all computing tasks that include shuffle operations. It can identify skewed data in advance and support users to specify how to process skewed data. It can optimize the problem of long shuffle processes caused by skewed data, reduce resource consumption, reduce long-tail tasks, and improve the computing efficiency of shuffle analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 This is a schematic diagram of the skew data identification and processing in the shuffle process of the present invention; Figure 2 This is the implementation flow of the tilted data identification and processing of the present invention. DETAILED DESCRIPTION
[0013] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings: The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments. The purpose and effect of the present invention will become more clear. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0014] Explanation of terms The relevant concepts of the present invention are as follows: Data skew: During the shuffle process, the amount of data distributed from the Map side to some Reduce partitions is much larger than that to other Reduce partitions. This causes the corresponding Reduce tasks to take longer to pull data, increases processing time, and slows down the overall task progress.
[0015] "Long-tail" tasks: Spark is a distributed in-memory computing framework. Among the many tasks running in parallel, a few can take significantly longer to execute than others, significantly extending the total execution time of the entire stage or job. These tasks, which take significantly longer than other tasks in the same batch, are called "long-tail tasks."
[0016] RSS (Remote Shuffle Service) shuffle: RSS shuffle is an efficient shuffle method that uses the RSS service to complete the shuffle process. This shuffle method has good stability and performance. The RSS service is a set of services with a master-slave architecture. The RSS Master is the master node responsible for managing the metadata of the shuffle data, and the RSSWorker is the slave node responsible for managing the specific shuffle data. It is responsible for managing the shuffle data of Spark tasks. The RSS shuffle method works by sending the same partition data to the remote RSS service. The RSS service merges the data belonging to the same reduce partition, and the reduce task directly reads the merged partition data from the RSS service.
[0017] Shuffle key: Shuffle key refers to the field used to partition data when shuffling data. When the Map task processes each row of data, it calculates which Reduce partition the row of data belongs to based on the hash value of the shuffle key.
[0018] Shuffle Writer: The component in the Map task responsible for writing shuffle data to the RSS service.
[0019] Shuffler Reader: The component in the Reduce task responsible for reading shuffle data from the RSS service.
[0020] Currently, Spark recommends using the RSS shuffle method, so we will explain the design ideas of the solution of the present invention based on the RSS shuffle process.
[0021] The RSS shuffle process is as follows Figure 1As shown, each Map task processes a data shard. After the Map task completes data processing, the ShuffleWriter sends the data belonging to each Reduce partition to the RSS Worker. After receiving the calculation results of the Map task, the ShuffleWriter caches the data according to the shuffle partition to which each data item belongs in the buffer. When the buffer is full, the data in the buffer is pushed to the RSS Worker. After receiving data from different Maps, the RSS Worker merges and writes it to the buffer. When the buffer is full, it overflows to the local disk. The Reduce task obtains the partition data location information from the RSSMaster and pulls the data belonging to its partition from the corresponding RSS Worker.
[0022] To address the data skew problem caused by the shuffle process, we propose a solution that identifies skewed data on the Map side. The key idea is to add a skewed data identification and processing module to the Map task. In the first phase, when writing the first several batches of data on the Map side, we first use a coarse-grained recognition method to identify the Reduce partitions to which the skewed shuffle keys belong. In the second phase, we use several batches of data written on the Map side to count which shuffle keys in the skewed Reduce partitions identified in the first phase are hot data, potentially causing data skew.
[0023] The detailed hotspot data identification and processing implementation process is as follows: Figure 2 As shown in the figure, in each Map task, after the data processing operator on the Map side processes the data of a single data shard, it is handed over to the ShuffleWriter component in the format of RDD[InternalRow] for processing. ShuffleWriter converts the received RDD into an iterator in the format of Product2[K,V], and first writes the data to the buffer of the corresponding Reduce partition. When the buffer is full, the data in the buffer is pushed to the RSS Worker.
[0024] Since the operators on the Map side are diverse, the hot data identification and processing module should be added to ShufleWriter. The identification and processing process is divided into three stages: Phase 1: Coarse-grained identification of skewed Reduce partitions: First, the data is divided into batches, each containing a fixed number of rows (defined as batch_size). When ShuffleWriter receives the first M batches of data and writes [K, V] data to the buffer, it first counts the number of rows belonging to each Reduce partition. If the number of rows in any Reduce partition within the first M batches exceeds a certain threshold (skewed_partition_rows_number_threshold) and is greater than a certain multiple (skewed_partition_factor) of the median number of rows across all partitions, the partition is considered skewed.
[0025] Phase 2: Fine-grained identification of hotspot shuffle keys: The first phase identified the Reduce partitions where the skewed data resides. In the second phase, we further identify the shuffle keys causing the skew within the skewed Reduce partitions. In the second phase, we use Map output data from batches (M+1) to (N) to identify skewed shuffle keys. Specifically, before writing data to the buffer, we count the total number of data entries for each shuffle key. After the Nth batch of data is sent, we identify shuffle keys whose total number of data entries exceeds a certain threshold (skewed_shufflekey_rows_number_threshold) and is greater than a certain multiple (skewed_partition_factor) of the median number of data entries for all shuffle keys as skewed.
[0026] Phase 3: Processing skewed data: After all skewed shuffle keys have been identified in the second phase, the third phase begins processing skewed data. In actual application scenarios, skewed data is mostly invalid data. Therefore, for identified skewed shuffle keys, by default, they are discarded according to the skewed data discard ratio and no longer participate in the shuffle process. Furthermore, the client can specify the discard ratio for skewed shuffle keys via a hint when issuing SQL. If the client specifies a discard ratio, the skewed shuffle keys are discarded according to the client's specified discard ratio. At the same time, the skewed shuffle key data is sampled and recorded, and the total number of all skewed shuffle keys in each Map task is counted and recorded in a statistical table.
[0027] The present invention identifies skewed data in advance and improves shuffle performance: Spark's native skewed partitions are calculated based on the statistical information of the shuffle data after the shuffle data is written on the Map side. At this time, the shuffle data has all been written to the RSS Worker. The shuffle process still consumes a large amount of disk and network IO, and the problem of long shuffle time cannot be effectively solved. The present invention proposes a method of first using a small amount of data to identify skewed partitions in the first few data processing batches at the beginning of the Map task's calculation process, and then further using a small amount of data to identify skewed shuffle keys. The method also supports users to specify how to handle skewed shuffle data, thereby improving the flexibility of using Spark for offline analysis in actual application scenarios, reducing the participation of invalid data in the shuffle process, saving resource consumption in the shuffle process, and reducing the time consumption of the shuffle process, thereby improving the execution efficiency of shuffle-type Spark computing tasks.
[0028] This invention is a general solution applicable to all computing tasks involving shuffle operations. It can identify skewed data in advance and support user-defined skewed data processing methods. This can optimize the time-consuming shuffle process caused by skewed data, reduce resource consumption, reduce "long-tail" tasks, and improve the computational efficiency of shuffle-related analysis tasks.
[0029] Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention shall be included in the scope of protection of the invention. All technical features in this embodiment may be freely combined according to actual needs.
[0030] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A Spark-based data tilt optimization method, characterized by: Add a tilt data recognition and processing module to the Map task, which is divided into three stages: In the first stage, coarse-grained identification of skewed Reduce partitions is used: When writing the first several batches of data on the Map side, the Reduce partitions to which the skewed shuffle keys belong are identified through coarse-grained identification. Phase 2: Fine-grained identification of hot shuffle keys: Find the shuffle key that causes the skew in the Reduce partition to which the skewed shuffle key belongs; Phase 3: Processing skewed data: Process all skewed shuffle keys identified in Phase 2.
2. The Spark-based data tilt optimization method according to claim 1, wherein: In the first phase, skewed Reduce partitions are identified at a coarse-grained level, which includes the following steps: Step A1: Divide the data into batches. Each batch contains a fixed number of data items, which is defined as batch_size. In step A2, when ShuffleWriter receives the first M batches of data and writes [K, V] data to the buffer, it first counts the number of entries in each Reduce partition. If the number of data entries in some Reduce partitions in the first M batches exceeds a certain threshold and is greater than a certain multiple of the median number of data entries in all partitions, it is considered a skewed partition.
3. The Spark-based data tilt optimization method according to claim 1, wherein: In the second phase, fine-grained identification of hotspot shuffle keys involves the following steps: Step B1: Based on the Reduce partition where the skewed data is located, the shuffle key causing the skew is further found in the skewed Reduce partition. In step B2, skewed shuffle keys are identified through the Map output data from batches M+1 to N. Specifically, before writing data to the buffer, the total number of data items for each shuffle key is counted. After the Nth batch of data is sent, shuffle keys whose total number of data items is greater than a certain threshold and greater than a certain multiple of the median number of data items of all shuffle keys are identified as skewed shuffle keys.
4. The Spark-based data tilt optimization method according to claim 1, wherein: In the third stage, the tilted data is processed, which includes the following steps: In step C1, based on all skewed shuffle keys identified in the second phase, skewed data processing begins. In actual application scenarios, most skewed data is invalid data. Therefore, for identified skewed shuffle keys, the skewed shuffle keys are discarded by default according to the skewed data discard ratio and no longer participate in the shuffle process. Step C2: When the client sends SQL, you can specify the discard ratio of the tilted shuffle key through a hint. If the client specifies the discard ratio, the tilted shuffle key is discarded according to the discard ratio specified by the client. In step C3, the tilted shuffle key data is sampled and recorded, and the total number of all tilted shuffle keys in each Map task is counted and recorded in the statistical table.