Shuffle sorting method, system, and storage medium

Abstract

The application discloses a shuffle sorting method, a system and a storage medium, and is used for a shuffle sorting algorithm based on a big data remote service, and improves the performance and stability of a big data computing framework. The application provides a shuffle sorting method, which is applied to a shuffle sorting system, the shuffle sorting system comprises a Map end, a ShuffleWorker component, a ShuffleSorter component and a Reduce end, and the method can comprise the following steps: outputting data to the ShuffleWorker component through the Map end; sequentially writing the data output by the Map end into a data block through the ShuffleWorker component; sorting the data in the data block through the ShuffleSorter component; and reading the sorted data in the data block through the Reduce end.

Description

Technical Field

[0001] This application relates to the field of big data computing, and in particular to a shuffle sorting method, system, and storage medium. Background Technology

[0002] In big data computing, after the Reduce phase pulls data from the Map phase, it sorts data with the same key value before performing further computations. During the sorting process on the Reduce phase, if the pulled data is large, the Reduce phase divides the data into multiple shards, sorts the data in each shard, and spills the sorted data to disk. Finally, the data from different shards on disk are merged and sorted one by one. This entire process involves multiple sorting and data merging operations, as well as multiple disk spill operations. This places high demands on the performance and stability of the big data computing framework. Summary of the Invention

[0003] This application provides a shuffle sorting method, system, and storage medium for using a shuffle sorting algorithm based on big data remote services to improve the performance and stability of big data computing frameworks.

[0004] This application provides a shuffle sorting method, which is applied to a shuffle sorting system. The shuffle sorting system includes a Map phase, a ShuffleWorker component, a ShuffleSorter component, and a Reduce phase. The method may include:

[0005] Data is output to the ShuffleWorker component through the Map terminal;

[0006] The ShuffleWorker component sequentially writes the data output from the Map end into the data block.

[0007] The data in the data block is sorted using the ShuffleSorter component.

[0008] The Reduce module reads the sorted data from the data block.

[0009] A second aspect of this application provides a shuffle sorting system, which includes a Map side, a ShuffleWorker component, a ShuffleSorter component, and a Reduce side;

[0010] The Map component is used to output data to the ShuffleWorker component.

[0011] The ShuffleWorker component is used to sequentially write the data output from the Map end into the data block;

[0012] The ShuffleSorter component is used to sort the data in the data block;

[0013] The Reduce end is used to read the sorted data in the data block.

[0014] A third aspect of this application provides a shuffle sorting system, which may include:

[0015] Memory containing executable program code;

[0016] A processor and a transceiver coupled to the memory;

[0017] The shuffle sorting system is used to perform the method described in the first aspect of this application.

[0018] In another aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a processor, cause the processor to perform the method described in the first aspect of this application.

[0019] Another aspect of this application discloses a computer program product that, when run on a computer, causes the computer to execute the method described in the first aspect of this application.

[0020] Another aspect of this application discloses an application publishing platform for publishing computer program products, wherein when the computer program product is run on a computer, the computer executes the method described in the first aspect of this application.

[0021] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0022] This application provides a shuffle sorting method applied to a shuffle sorting system, which includes a Map phase, a ShuffleWorker component, a ShuffleSorter component, and a Reduce phase. The method includes: outputting data from the Map phase to the ShuffleWorker component; sequentially writing the data output from the Map phase into data blocks via the ShuffleWorker component; sorting the data in the data blocks via the ShuffleSorter component; and reading the sorted data from the data blocks via the Reduce phase. Data generated by the Map phase does not need to be sorted and spilled to disk locally; it is directly sent to a remote ShuffleWorker via the network. This mechanism solves the problem of existing technologies where large amounts of Map phase data are spilled to local disks, putting significant pressure on local disk I / O and affecting the stability of overall computation. The Reduce phase only needs to read data from specific machines, without needing to connect to all the machines hosting the Map phases. This avoids creating too many network connections when there are many Map phases, reducing the impact of network and machine failures on the stability of data reading. The Reduce side no longer needs to perform single-point sorting during data reading. The reading order is already globally ordered, so there is no need to sort on the Reduce machine. This reduces the requirements for computing resources and avoids large amounts of data being spilled to disk multiple times, significantly improving the computing stability of a single machine. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments and the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application, and other drawings can be obtained based on these drawings.

[0024] Figure 1 A schematic diagram of the Map and Reduce phases of big data computing;

[0025] Figure 2 A schematic diagram of the framework of the shuffle sorting system used in the embodiments of this application;

[0026] Figure 3 A schematic diagram illustrating the differences between the serial model and the pipeline model;

[0027] Figure 4 This is a schematic diagram of one embodiment of the shuffle sorting method in this application;

[0028] Figure 5AThis is a schematic diagram illustrating the ShuffleWorker multi-machine switching mechanism used in an embodiment of this application;

[0029] Figure 5B This is a schematic diagram illustrating the multiple computing engine frameworks supported by embodiments of this application;

[0030] Figure 5C This is a schematic diagram showing the simultaneous operation of ShuttleWorker and ShuffleSorter in an embodiment of this application;

[0031] Figure 5D This is a schematic diagram illustrating how the ShuffleSorter component sorts data and adds indexes in an embodiment of this application.

[0032] Figure 5E This is a flowchart illustrating the process of querying data within the entire computing engine and the Shuffle sorting algorithm model of this application in an embodiment of the present application;

[0033] Figure 6 This is a schematic diagram of the shuffle sorting system in an embodiment of this application;

[0034] Figure 7 This is a schematic diagram of another embodiment of the shuffle sorting system in this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions of the embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. All embodiments based on the present application should fall within the scope of protection of the present application.

[0036] The following is a brief explanation of some of the terms used in this application:

[0037] Map, an operator in big data computing models, splits data according to different keys.

[0038] Reduce, an operator in big data computing models, is a computing process that groups data by the same key.

[0039] Spark is an open-source big data computing engine. Its full name is Apache Spark.

[0040] Flink is an open-source real-time compute engine for big data.

[0041] Trino is an open-source ad-hoc compute engine for big data.

[0042] Shuffle refers to the data shuffling and exchange process in big data computing.

[0043] Shuffle-Sort refers to the sorting computation during the shuffle and exchange data process of big data computing.

[0044] ShuffleWorker, a component introduced in this application, is responsible for collecting shuffle data.

[0045] ShuffleSorter, a component introduced in this application, is responsible for sorting operations (A module used to sort data, imported by this patent).

[0046] Checksum, a data verification code, is used to check the correctness of data reading and writing.

[0047] In the field of big data computing, the Spark computing engine has become the de facto standard computing engine after more than a decade of development. From the earliest Spark 1.0 version to the current Spark 3.0 version, and with each iteration of Spark, its shuffle and sorting algorithms have been continuously upgraded, from the initial hash shuffle to the now most commonly used sorting shuffle. The current Spark shuffle algorithm, released starting with Spark 2.0, is relatively mature and stable, and has been widely adopted in the industry.

[0048] The main workflow of big data computation in Spark, Flink, or Trino computing engine frameworks can be abstracted into two stages: the Map stage and the Reduce stage. The Map stage partitions the data according to different key values, and the Reduce stage aggregates the data according to the same key value before performing computation, such as... Figure 1 The diagram shown illustrates the Map and Reduce phases of big data computing.

[0049] exist Figure 1 As shown, the Map task divides the input data into two partitions, A and B, based on the key value, which are then read by two Reduce tasks.

[0050] After the Reduce module pulls data from the Map module, it sorts data with the same key value before performing further computations. During the sorting process on the Reduce module, if the pulled data is large, the Reduce module divides the data into multiple shards, sorts the data in each shard, and spills the sorted data to disk. Finally, the data from different shards on disk are merged and sorted one by one. The entire process involves multiple sorting and data merging operations, as well as multiple disk spill operations.

[0051] Existing technologies have shortcomings in both the stability and performance of the computation process. These shortcomings will be analyzed from these two aspects below:

[0052] pass Figure 1 As shown in the diagram, each Map task first writes its generated data to the local disk, writing to different files according to partitions during the writing process. Each downstream Reduce task reads data from a single partition. The Reduce task sends network requests to all upstream Map tasks to read a specific file from the machine where the Map tasks reside. After fetching the corresponding partition data, the Reduce task sorts it locally and then provides the sorted data to the upper-layer application for computation.

[0053] The following are the points that affect stability during this process:

[0054] 1. Each Map task writes data to the local disk. If some Map tasks need to write a large amount of data, it will put a lot of pressure on the disk of the machine in a short period of time, causing the input / output (IO) to be full, and the entire machine will be slowed down. This will not only affect the operation of the task itself, but also affect other tasks running on the machine.

[0055] 2. Each Reduce task needs to request data from the machine where each Map task is located upstream. If a machine or network connection fails, it will cause the data retrieval to fail, which will in turn cause the Reduce task to fail.

[0056] 3. After the Reduce task retrieves the required partition data, it sorts the data locally. The entire sorting process has high memory requirements. If the memory setting for the Reduce task is too small, a memory overflow error will occur, causing the Reduce task to fail.

[0057] The following are some of the ways in which the existing technical solution affects performance:

[0058] 1. Each Reduce task needs to request data from all upstream Map tasks, requiring multiple network requests. Under normal network and Map machine conditions, each network request consumes at least 30 milliseconds. Furthermore, in actual production clusters, both the network and machines are quite busy, and network response times may exceed 10 seconds. If there are too many upstream Map tasks, such as more than 10,000, the network request time will be significant, impacting task computation performance.

[0059] 2. Each Reduce task needs to request a portion of the output data from multiple Map tasks, and the output data of one Map task is read by multiple Reduce tasks. From the perspective of the Map task, the order of each Reduce task's request is random. Therefore, each data read involves randomly reading fragmented data. Given the inherent characteristics of the disk, the performance difference between reading fragmented random data and reading sequentially from a complete block of data is significant, at least tenfold. Therefore, this fragmented random approach severely degrades the computational performance of the task.

[0060] 3. Reduce tasks sort data only after the Shuffle process has read the data. This process is sequential, and in distributed systems, this sequential process can become a performance bottleneck. Therefore, from a computational workflow perspective, this approach in the current technical solution has a significant impact on performance.

[0061] 4. After the Reduce function reads the Shuffle data, it needs to sort it locally. If the data volume is large, multiple disk writes and multiple merge sorting processes are required. This single-point sorting computation places high demands on machine resources and has a significant impact on performance. In actual online business operations, it is common to encounter situations where the sorting time of a single Reduce task is more than ten times that of other computation steps.

[0062] In summary, the current solution suffers from serious deficiencies in both stability and computational performance. For big data computing, this technology results in a significant waste of computing resources, hindering the healthy development of online businesses.

[0063] like Figure 2 The diagram shown is a schematic representation of the framework of the shuffle sorting system used in the embodiments of this application. Figure 2 This is a shuffle sorting algorithm model based on remote services. The "remote service" refers to the service that initiates the data processing, i.e., sending the data to a remote server. Existing technologies utilize local disks for data processing. In this embodiment, the service that initiates the data processing is a ShuffleWorker component and / or a ShuffleSorter component. This algorithm model aims to improve stability and performance, proposing corresponding solutions to the aforementioned shortcomings. The Map side can also be referred to as a Map task, and the Reduce side can also be referred to as a Reduce task.

[0064] First, let me introduce the main idea of ​​the shuffle sorting algorithm model in this application:

[0065] 1. Convert random read / write operations on fragmented disks into sequential read / write operations.

[0066] Current technical solutions involve fragmented random data read / write operations on disk during both the Map task execution and Shuffle phases. From the perspective of disk physical characteristics, the performance of fragmented random data read / write is at least an order of magnitude slower than sequential data read / write. Therefore, transforming fragmented random read / write into sequential read / write becomes an important optimization direction for improving performance.

[0067] 2. Transform the shuffle and sorting processes from a serial model to a pipelined execution model.

[0068] like Figure 3 The diagram shown illustrates the differences between the serial model and the pipeline model. Figure 3 As shown in the analysis, in terms of efficiency comparison between the serial and pipelined execution modes, the pipelined execution mode has the two processes overlapping most of the time, so the total execution time is significantly shorter than that of the serial execution mode.

[0069] 3. Convert single-point large-scale data sorting calculations into parallel computing.

[0070] The key technical characteristic of big data is its use of distributed parallel technology to process massive amounts of data that cannot be handled by a single machine. However, existing technical solutions still require such heavyweight computational operations to be performed on a single machine. Therefore, another advantage of this application is that it decomposes the deep single-point serial computation pressure by understanding the essence of big data computing. Furthermore, since the main operation of the Reduce task is to merge data with the same key value, a challenge in parallelization is how to correctly merge the results of distributed parallel computation.

[0071] This application introduces a remote service component called ShuffleWorker and / or ShuffleSorter. ShuffleWorker is primarily responsible for aggregating Shuffle data sent from the Map phase, sequentially writing data from the same partition across different Map phases into different data blocks. These data blocks can be temporarily stored in memory or on disk. Optionally, data blocks are split into fixed-size blocks, with a default threshold of 128MB. When a data block is written to memory or disk, it is detected by the corresponding ShuffleSorter component. The ShuffleSorter component then sorts the data in that block and writes the sorted block back to memory or disk. The Reduce task reads all data blocks from the corresponding partition. Reading data blocks is a sequential process, and since each data block is ordered, the data read by the Reduce task no longer needs to be sorted.

[0072] The technical solution of this application will be further described below by way of embodiments, such as... Figure 4 The diagram shown is a schematic representation of an embodiment of the shuffle sorting method in this application. The method is applied to a shuffle sorting system, which includes a Map side, a ShuffleWorker component, a ShuffleSorter component, and a Reduce side. This method embodiment may include:

[0073] 401. Output data to the ShuffleWorker component through the Map terminal.

[0074] Combination Figure 2 The example shown is for illustration. Figure 2The diagram shows three Map tasks, each outputting two partitions of data: P0 and P1. Two ShuffleWorkers are responsible for collecting the data from P0 and P1 respectively, meaning each Map task sends the P0 and P1 partition data to the two ShuffleWorkers respectively.

[0075] Optionally, the step of outputting data to the ShuffleWorker component through the Map terminal may include: sending a token request to the ShuffleWorker component through the Map terminal; determining a token and sending the token to the Map terminal based on the token request, provided that the remaining resources of the ShuffleWorker are greater than or equal to a preset resource threshold; sending flow control exception information to the Map terminal if the remaining resources of the ShuffleWorker are less than the preset resource threshold; receiving the token or flow control exception information sent by the ShuffleWorker component through the Map terminal; outputting data carrying the token to the ShuffleWorker component; or, sending the token request to the ShuffleWorker component again if a preset time period has elapsed.

[0076] For example, when a Map task sends data to a ShuffleWorker, various abnormal situations may occur, such as ShuffleWorker traffic overload, causing the ShuffleWorker service to be too busy and unable to respond in a timely manner. This application embodiment introduces a flow control mechanism, the specific logic of which is as follows:

[0077] Before sending a data packet to the ShuffleWorker, the Map task needs to request a token pair from the ShuffleWorker. Optionally, the token contains two numbers representing its key-value pair. This key-value pair is included when sending the data packet to the ShuffleWorker, which will then verify it. If the ShuffleWorker's resources are strained, it will return a flow control exception indication to the Map task. The Map task will then wait for a fixed period of time before attempting to reacquire the token, and only send the data after successfully obtaining it.

[0078] Optionally, the step of outputting data to the ShuffleWorker component through the Map terminal may include: outputting data to a first ShuffleWorker through the Map terminal, and outputting the data to a second ShuffleWorker in the event of a failure of the first ShuffleWorker, wherein the ShuffleWorker component includes the first ShuffleWorker and the second ShuffleWorker.

[0079] For example, when a Map task sends data to a ShuffleWorker, various abnormal situations may occur, such as the ShuffleWorker machine or service failing, causing the Map task to be unable to connect and send data. In this embodiment, a ShuffleWorker multi-machine switching mechanism can be used to solve this problem. Figure 5A The image shown is a schematic diagram illustrating the ShuffleWorker multi-machine switching mechanism used in an embodiment of this application. Figure 5A As shown, the Map task initially sends data to ShuffleWorker A. When ShuffleWorker A encounters a problem and the connection is broken, it attempts to send data to ShuffleWorker B. Similarly, if ShuffleWorker B also malfunctions and cannot provide service, the Map task switches to ShuffleWorker C. This polling switching mechanism ensures that the overall Shuffle service provides stable service even in complex environments.

[0080] Optionally, the ShuffleWorker component, and / or the ShuffleSorter component, is a remote service.

[0081] Optionally, the method can be applied to the computation engine frameworks of Spark, Flink, and / or Trino.

[0082] For example, such as Figure 5B The diagram shown illustrates the various computing engine frameworks supported by this application's embodiments. The computing engine frameworks supported by this application's embodiments are Spark, Flink, and / or Trino.

[0083] 402. Using the ShuffleWorker component, the data output from the Map end is sequentially written into the data block.

[0084] Combination Figure 2The example shown illustrates how each ShuffleWorker, upon receiving data, merges the data and writes it sequentially into the corresponding data blocks. Taking data P0 as an example, it is divided into three data blocks: P0-1 to P0-3. The order in which the data blocks are generated can be referenced by their suffix numbers; for example, P0-1 is generated first, followed by P0-2, and finally P0-3. Optionally, to avoid multiple disk I / O operations, data blocks are stored in memory whenever possible.

[0085] Optionally, the step of sequentially writing the data output from the Map end into data blocks through the ShuffleWorker component may include: sequentially writing the data output from the Map end into different data blocks through the ShuffleWorker component.

[0086] Optionally, the data blocks are stored in memory or on a disk.

[0087] Optionally, the step of sequentially writing the data output from the Map end into the data block through the ShuffleWorker component may include: when receiving data carrying the token through the ShuffleWorker component, verifying whether the token is valid; if the token is valid, releasing the token to the available token pool, and sequentially writing the data into the data block.

[0088] Optionally, the ShuffleWorker component includes a token cache pool, which includes the available token pool and the unavailable token pool. The available token pool stores at least one of the tokens that the ShuffleWorker component has not issued and the tokens that have been issued and released. The unavailable token pool stores the tokens that the ShuffleWorker component has issued but not released.

[0089] For example, on the ShuffleWorker side, a token cache pool is used to record available tokens, as well as tokens that have already been issued and are currently in use. When the ShuffleWorker receives a data packet with a token, it first verifies whether the token in the data packet is valid, i.e., whether it is a token that has been issued and is currently in use. If the verification is valid, the token is released to the available token pool, and the data packet is written to the data block. If the ShuffleWorker issues too many tokens that are currently in use, or if too much memory is used, exceeding the set threshold, new token requests can be postponed, and a flow control exception indication can be returned to the Map task. Token requests can then be re-entered after a fixed period of time.

[0090] 403. Sort the data in the data block using the ShuffleSorter component.

[0091] Combination Figure 2 The example shown illustrates how the ShuffleSorter, upon sensing the data block written by the ShuffleWorker, immediately begins sorting the data within that block. For the same partition of data (e.g., P0), at any given moment, one ShuttleWorker might be generating data, while multiple ShuffleSorters are sorting the already generated data. For example... Figure 5C The diagram shown is a schematic of ShuttleWorker and ShuffleSorter working simultaneously in an embodiment of this application.

[0092] exist Figure 5C In the diagram, in the first row, the fully shaded data block P0-1 is the data block that the ShuffleWorker has already written, the gradually shaded data block P0-2 is the data block being written, and the unshaded data block P0-3 is the data block to be written. In the second row, the fully shaded data block P0-1 is the data block that the ShuffleSorter is sorting; in the third and fourth rows, the slash-shaded data blocks P0-2 and P0-3 are the data blocks to be sorted. Overall, the writing and sorting processes of data blocks overlap concurrently, thus significantly improving overall computational efficiency.

[0093] 404. Read the sorted data in the data block through the Reduce terminal.

[0094] Combination Figure 2 The example shown illustrates how the Reduce task reads sorted data from data blocks P0-1, P0-2, P0-3, P1-1, P1-2, and P1-3.

[0095] Optionally, reading the sorted data in the data block through the Reduce end may include: using the loser tree algorithm model of the multi-way merge sort algorithm to read the sorted data in the data block through the Reduce end.

[0096] For example, when a Reduce task reads a partition, such as P0-1 to P0-3, these three data blocks are ordered internally, but the data between each block may not be globally ordered. In this embodiment, the Reduce side selects the loser tree algorithm model of the multi-way merge sort algorithm, which ensures that the reading order is globally ordered. The time complexity of this algorithm is nlog(n), but it is highly optimized for disk I / O, and the overall sequential data reading ensures efficient read operations. More importantly, this natural ordering of the read data avoids multiple disk spills during the local sorting process on the Reduce side, eliminating the pressure on the Reduce side's memory and disk I / O, and improving the resource utilization and overall stability of the task.

[0097] Optionally, the Map endpoint includes a first Map and a second Map, and the data includes first data and second data. Outputting data to the ShuffleWorker component through the Map endpoint may include: calculating a first reference checksum for the first data through the first Map, and calculating a second reference checksum for the second data through the second Map; when the first data and the second data are in the same partition, summing the first reference checksum for the first data and the second reference checksum for the second data to obtain a reference accumulated checksum; outputting the first data to the ShuffleWorker component through the first Map, outputting the second data to the ShuffleWorker component through the second Map; and sending the reference accumulated checksum to the ShuffleWorker component.

[0098] The step of sequentially writing the data output from the Map end into the data block through the ShuffleWorker component may include: writing the first data, the second data, and the accumulated check value into the data block through the ShuffleWorker component;

[0099] The step of reading sorted data from the data block via the Reduce endpoint may include: reading sorted first data, second data, and the reference accumulated checksum from the data block via the Reduce endpoint; calculating a first current checksum for the first data and a second current checksum for the second data; accumulating the first current checksum and the second current checksum to obtain a current accumulated checksum; and determining whether the read first data and second data are accurate based on the current accumulated checksum and the reference accumulated checksum.

[0100] Optionally, determining whether the read first data and second data are accurate based on the current accumulated checksum and the reference accumulated checksum may include:

[0101] If the current accumulated check value is the same as the reference accumulated check value, it is determined that the first data and the second data read are accurate;

[0102] If the current accumulated check value is different from the reference accumulated check value, it is determined that the first data and the second data read are abnormal.

[0103] For example, in distributed computing, data correctness is paramount. When data enters the shuffle sorting system, how can we ensure that the data read last matches the data written last? In this application, a checksum mechanism can be used to guarantee data correctness. The specific logic is as follows: Before sending each piece of data to the ShuffleWorker, the Map task calculates a checksum value. The checksum value for each partition is accumulated in memory. After all data for a partition has been sent, the accumulated checksum value for that partition is sent to the ShuffleWorker, which writes the received checksum value to the data file. Similarly, the Reduce task calculates and accumulates the checksum value for each piece of data it reads. Once all data has been read, the checksum value is compared with the checksum value sent by the Map task. If any data is missing or duplicated, the checksum comparison result will show an anomaly.

[0104] Optionally, the checksum value can be sent along with the data via ShuffleWorker.

[0105] Optionally, the Checksum value is carried in the last data packet of the data.

[0106] Optionally, sorting the data in the data block using the ShuffleSorter component may include: sorting the data in the data block and adding an index using the ShuffleSorter component;

[0107] The step of reading the sorted data in the data block through the Reduce terminal may include: reading the index of the sorted data in the data block through the Reduce terminal; and determining whether the data is the required data based on the index.

[0108] For example, the ShuffleSorter component can add indexes to the data during the sorting process, such as simple indexes like Min-Max. This allows downstream Reduce tasks to filter and initially screen the sorted data blocks, reducing the amount of data that needs to be read and thus improving overall performance. Figure 5D The diagram illustrates how the ShuffleSorter component sorts data and adds indexes in an embodiment of this application. It can be understood that before the Reduce task reads the sorted data, it first reads the corresponding index to determine whether the corresponding data needs to be read or which part of the data needs to be read. Figure 5D In the Reduce task's data reading pipeline, dashed lines indicate that only part of the data needs to be read, or no data needs to be read at all.

[0109] Optionally, the following example illustrates the detailed data flow process in this application's big data computing process:

[0110] Big Data Query SQL:

[0111] SELECT u.name, o.orderId FROM user u JOIN order o ON u.uid = o.uid; / / Retrieves all user names and their corresponding order numbers from two tables: a user table and an order table.

[0112] Performing a join query on the user and order tables is a typical shuffle sorting scenario in big data computing engines. The user table is Table 1, and the order table is Table 2. Example data from both tables is as follows:

[0113] 1 Zhang San 2 Li Si

[0114] Table 1

[0115] 1 101 1 102 2 103 1 104 3 105 2 106 1 107

[0116] Table 2

[0117] like Figure 5E The diagram shown illustrates the process of querying data within the entire computing engine and the Shuffle sorting algorithm model of this application, as described in an embodiment of this application. Figure 5EAs shown, Map1 and Map2 read data from two tables, representing two join data streams. Each Map outputs a partition of data. ShuffleWorker1 generates a data block, which is then sorted by ShuffleSorter1. ShuffleWorker2 generates two data blocks, which are similarly sorted by ShuffleSorter2. Finally, in the Reduce phase, both data streams are sorted. For example, when joining data with uid 1, the user table first reads the first value <1, Zhang San>, then reads data from the order table: <1, 101>, <1, 102>, <1, 104>, <1, 107>. Since the data is already sorted, the uids of these values ​​match <1, Zhang San>, and the join succeeds. Both sides continue reading data, and so on.

[0118] The beneficial results of the technical solution of this application are compared with those of current technical solutions, and are described separately in terms of stability and performance, as follows:

[0119] stability:

[0120] 1. The shuffle data generated by the Map task does not need to be sorted and spilled to disk locally; it is sent directly to the remote ShuffleWorker over the network. This mechanism solves the problem of existing technologies where a large amount of Map data is spilled to the local disk, putting significant pressure on the local disk I / O and affecting the stability of the overall machine's computing.

[0121] 2. Reduce tasks only need to read data from specific machines, eliminating the need to connect to all the machines hosting Map tasks. This avoids creating too many network connections when there are many Map tasks, reducing the impact of network and machine failures on the stability of data reading.

[0122] 3. The Reduce task no longer needs to perform single-point sorting during the data reading process. The data reading order is already naturally globally ordered, so there is no need to sort on the Reduce machine. This reduces the requirements for computing resources and avoids the impact of multiple spills of large amounts of data to disk, significantly improving the computing stability of a single machine.

[0123] Computational performance:

[0124] 1. Reduce does not need to request all the machines where Map tasks are located, reducing unnecessary network overhead and improving network transmission efficiency.

[0125] 2. Data read and write operations are transformed from random fragmented read and write to sequential read and write, making full use of the advantages of sequential read and write on the disk to improve read and write performance.

[0126] 3. By using a pipeline model, the shuffle and sorting processes are transformed from serial computation to parallel overlapping computation, making full use of distributed technology to improve computational efficiency.

[0127] 4. The sorting process is transformed from single-point sorting to distributed sorting. At the same time, merging the sorted results avoids multiple disk I / O operations, which can significantly reduce the sorting time.

[0128] This application proposes for the first time a shuffle and sorting model based on remote services, which unifies and integrates the big data shuffle and sorting processes. The main core innovations are as follows:

[0129] Overall, this paper proposes using remote services to solve problems that are difficult for big data computing systems to handle on their own. This approach is more flexible, pluggable, and inherently possesses the property of separating storage and computation. It fully utilizes external resources to address the complex problems of the system itself.

[0130] This application addresses inherent problems with the Shuffle process, such as random fragmented read / write and excessive network connections. The solution employs a merging strategy, consolidating partitioned data distributed across multiple upstream Map tasks into fixed-size data blocks. This not only resolves fragmented read / write issues but also mitigates the stability and performance problems caused by the excessive network connections required for downstream Reduce tasks to read data. To ensure data integrity and the stability of remote services, this application creatively introduces mechanisms such as checksums, flow control, and multi-machine switching.

[0131] This application's solution to the sorting problem employs a divide-and-conquer strategy. It breaks down the sorting computation, which places immense pressure on a single point in big data computing, into multiple smaller problems of equal scale. These smaller problems are then sorted concurrently in a distributed manner, and finally, a multi-way merge algorithm is used to aggregate the results. This cleverly avoids the stability and performance issues present in current technical solutions.

[0132] Through the above innovative solutions, the big data remote shuffle sorting algorithm of this application utilizes the faster characteristics of disk merge sequential read and write and the divide-and-conquer approach to solve the lack of computing power for single-point sorting. It can effectively solve the thorny problems in existing technical solutions and improve the performance and stability of the big data computing framework.

[0133] like Figure 6 The diagram shown is a schematic of a shuffle sorting system in an embodiment of this application. The shuffle sorting system includes a Map component 601, a ShuffleWorker component 602, a ShuffleSorter component 603, and a Reduce component 604.

[0134] Map component 601 is used to output data to ShuffleWorker component 602;

[0135] ShuffleWorker component 602 is used to sequentially write the data output by Map end 601 into data blocks;

[0136] ShuffleSorter component 603 is used to sort the data in the data block;

[0137] Reduce port 604 is used to read the sorted data in the data block.

[0138] Optionally, the ShuffleWorker component 602 is used to sequentially write data output from the Map end 601 into different data blocks.

[0139] Optionally, the Reduce side 604 is specifically used for the loser tree algorithm model using the multi-way merge sort algorithm to read the sorted data in the data block.

[0140] Optionally, the data blocks are stored in memory or on a disk.

[0141] Optionally, the Map side 601 includes a first Map and a second Map, and the data includes first data and second data;

[0142] Map component 601 is specifically used to calculate a first reference check value for the first data through the first Map and a second reference check value for the second data through the second Map; when the first data and the second data are in the same partition, the first reference check value of the first data and the second reference check value of the second data are accumulated to obtain a reference accumulated check value; the first data is output to the ShuffleWorker component 602 through the first Map and the second data is output to the ShuffleWorker component 602 through the second Map; and the reference accumulated check value is sent to the ShuffleWorker component 602.

[0143] ShuffleWorker component 602 is specifically used to write the first data, the second data, and the accumulated check value into a data block;

[0144] Reduce terminal 604 is specifically used to read the sorted first data, second data, and reference accumulated check value in the data block; calculate the first current check value of the first data and the second current check value of the second data; accumulate the first current check value and the second current check value to obtain the current accumulated check value; and determine whether the read first data and second data are accurate based on the current accumulated check value and the reference accumulated check value.

[0145] Optionally, the Reduce side 604 is specifically used to ensure that the first and second data read are accurate when the current accumulated checksum is the same as the reference accumulated checksum; and to ensure that the first and second data read are abnormal when the current accumulated checksum is different from the reference accumulated checksum.

[0146] Optionally, the Map side 601 is specifically used to send a token request to the ShuffleWorker component 602 through the Map side 601;

[0147] ShuffleWorker component 602 is specifically used to determine a token and send the token to Map terminal 601 when the remaining resources of ShuffleWorker are greater than or equal to a preset resource threshold, based on the token request; and to send flow control exception information to Map terminal 601 when the remaining resources of ShuffleWorker are less than the preset resource threshold.

[0148] Map terminal 601 is specifically used to receive the token or flow control exception information sent by ShuffleWorker component 602; output data carrying the token to ShuffleWorker component 602; or, if a preset time period is exceeded, send the token request to ShuffleWorker component 602 again.

[0149] Optionally, the ShuffleWorker component 602 is specifically used to verify whether the token is valid when receiving data carrying the token through the ShuffleWorker component 602, and if the token is valid, release the token to the available token pool and sequentially write the data into the data block.

[0150] Optionally, the ShuffleWorker component 602 includes a token cache pool, which includes the available token pool and the unavailable token pool. The available token pool stores at least one of tokens that the ShuffleWorker component 602 has not issued and tokens that have been issued and released. The unavailable token pool stores tokens that the ShuffleWorker component 602 has issued but not released.

[0151] Optionally, the Map side 601 is specifically used to output data to the first ShuffleWorker, and in the event of a failure of the first ShuffleWorker, to output the data to the second ShuffleWorker. The ShuffleWorker component 602 includes the first ShuffleWorker and the second ShuffleWorker.

[0152] Optionally, the ShuffleWorker component 602 is specifically used to sort the data in the data block and add indexes;

[0153] Map terminal 601 is specifically used to read the index of the sorted data in the data block; and determine whether the data is the required data based on the index.

[0154] Optionally, the shuffle sorting system is applied to the computation engine frameworks of Spark, Flink, and / or Trino.

[0155] Optionally, ShuffleWorker component 602 and / or ShuffleSorter component 603 serve remotely.

[0156] like Figure 7 The diagram shown is a schematic representation of another embodiment of the shuffle sorting system in this application, which may include:

[0157] Memory 701 storing executable program code;

[0158] A processor 702 and a transceiver 703 are coupled to the memory 701;

[0159] The processor 702 and transceiver 703 are used to perform the above-mentioned tasks. Figure 4 The steps in the illustrated embodiment will not be repeated here.

[0160] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A shuffle sorting method, characterized by, The method is applied to a shuffle sorting system, which includes a Map phase, a ShuffleWorker component, a ShuffleSorter component, and a Reduce phase. The method includes: Data is output to the ShuffleWorker component through the Map terminal; The ShuffleWorker component sequentially writes the data output from the Map end into the data block. The data in the data block is sorted using the ShuffleSorter component. The Reduce module reads the sorted data from the data block. The Map terminal includes a first Map and a second Map, and the data output by the Map terminal includes first data and second data. The step of outputting data to the ShuffleWorker component through the Map terminal includes: The first reference check value of the first data is calculated using the first Map, and the second reference check value of the second data is calculated using the second Map. When the first data and the second data are in the same partition, the first reference check value of the first data and the second reference check value of the second data are summed to obtain a reference summed check value. The first data is output to the ShuffleWorker component through the first Map, and the second data is output to the ShuffleWorker component through the second Map; and the reference accumulated check value is sent to the ShuffleWorker component. The step of sequentially writing the data output from the Map end into the data block through the ShuffleWorker component includes: The ShuffleWorker component is used to write the first data, the second data, and the reference accumulated check value into a data block. The step of reading the sorted data from the data block through the Reduce end includes: Through the Reduce end, the sorted first data, second data, and reference accumulated check value are read from the data block; the first current check value of the first data and the second current check value of the second data are calculated; the first current check value and the second current check value are accumulated to obtain the current accumulated check value; based on the current accumulated check value and the reference accumulated check value, it is determined whether the read first data and second data are accurate.

2. The method of claim 1, wherein, The step of sequentially writing the data output from the Map end into the data block through the ShuffleWorker component includes: The ShuffleWorker component sequentially writes the data output from the Map end into different data blocks.

3. The method of claim 1, wherein, The step of reading the sorted data from the data block through the Reduce end includes: Through the Reduce end, the loser tree algorithm model of the multi-way merge sort algorithm is used to read the sorted data in the data block.

4. The method according to any one of claims 1-3, characterized in that, The data blocks are stored in memory or on disk.

5. The method of claim 1, wherein, The step of determining whether the read first data and second data are accurate based on the current accumulated check value and the reference accumulated check value includes: If the current accumulated check value is the same as the reference accumulated check value, it is determined that the first data and the second data read are accurate; If the current accumulated check value is different from the reference accumulated check value, it is determined that the first data and the second data read are abnormal.

6. The method according to any one of claims 1-3, characterized in that, The step of outputting data to the ShuffleWorker component through the Map terminal includes: The Map client sends a token request to the ShuffleWorker component. The ShuffleWorker component determines a token based on the token request, and sends the token to the Map terminal if the remaining resources of the ShuffleWorker are greater than or equal to a preset resource threshold; if the remaining resources of the ShuffleWorker are less than the preset resource threshold, it sends flow control exception information to the Map terminal. The Map terminal receives the token or flow control exception information sent by the ShuffleWorker component; outputs data carrying the token to the ShuffleWorker component; or, if a preset time period is exceeded, sends the token request to the ShuffleWorker component again.

7. The method of claim 6, wherein, The step of sequentially writing the data output from the Map end into the data block through the ShuffleWorker component includes: When data carrying the token is received through the ShuffleWorker component, the validity of the token is verified. If the token is valid, the token is released to the available token pool, and the data is sequentially written into the data block.

8. The method of claim 7, wherein, The ShuffleWorker component includes a token cache pool, which includes an available token pool and an unavailable token pool. The available token pool stores at least one of tokens that the ShuffleWorker component has not issued and tokens that have been issued and released. The unavailable token pool stores tokens that the ShuffleWorker component has issued but not released.

9. The method of any one of claims 1-3, wherein, The step of outputting data to the ShuffleWorker component through the Map terminal includes: Data is output to the first ShuffleWorker through the Map terminal. If the first ShuffleWorker fails, the data is output to the second ShuffleWorker. The ShuffleWorker component includes the first ShuffleWorker and the second ShuffleWorker.

10. The method of any one of claims 1-3, wherein, The step of sorting the data in the data block using the ShuffleSorter component includes: The ShuffleSorter component is used to sort the data in the data block and add indexes. The step of reading the sorted data from the data block through the Reduce end includes: The Reduce module reads the index of the sorted data in the data block; based on the index, it determines whether the data is the required data.

11. The method of any one of claims 1-3, wherein, The method is applied to the computation engine frameworks Spark, Flink, and / or Trino.

12. The method according to any one of claims 1-3, characterized in that, The ShuffleWorker component, and / or the ShuffleSorter component, are remote services.

13. A shuffle sorting system characterized by, The shuffle sorting system includes a Map phase, a ShuffleWorker component, a ShuffleSorter component, and a Reduce phase; the Map phase includes a first Map and a second Map. The Map component is used to output data to the ShuffleWorker component, the data including first data and second data; it is also used to calculate a first reference check value of the first data through the first Map, and to calculate a second reference check value of the second data through the second Map. When the first data and the second data are in the same partition, the first reference check value of the first data and the second reference check value of the second data are added together to obtain the reference accumulated check value; The first data is output to the ShuffleWorker component through the first Map, and the second data is output to the ShuffleWorker component through the second Map; and the reference accumulated check value is sent to the ShuffleWorker component. The ShuffleWorker component is used to sequentially write the data output from the Map end into a data block; it is also used to write the first data, the second data, and the reference accumulated check value into the data block. The ShuffleSorter component is used to sort the data in the data block; The Reduce end is used to read the sorted data in the data block; it is also used to read the sorted first data, the second data, and the reference accumulated check value in the data block. Calculate the first current check value of the first data, and calculate the second current check value of the second data; The first current check value and the second current check value are summed to obtain the current accumulated check value; Based on the current accumulated check value and the reference accumulated check value, determine whether the read first data and second data are accurate.

14. A shuffle sorting system characterized by, include: Memory containing executable program code; A processor and a transceiver coupled to the memory; The shuffle sorting system is used to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium comprising instructions that, when executed on a processor, cause the processor to perform the method as claimed in any one of claims 1-12.

Images