Method, system and storage medium for performing shuffle-reduce operations

Abstract

The shuffle-reduce operation receives as input the files sorted and written by different map tasks, obtains batches of data from each input file, merges and sorts the batches of data to form large unified data segments. The shuffle-reduce operation is applied to the unified data segments to produce output data. The shuffle-reduce operation includes a commutative reduction operation that provides an output data amount that is significantly smaller than the input data amount. The output data is written to the memory. The process is repeated for different batches of data until the data from each input file is completely consumed and the output data has been fully formed. The shuffle-reduce operation greatly reduces the size of data that needs to be read by the reduce task in the shuffle operation, thereby significantly reducing the input / output overhead and the total execution time.

Description

Technical Field

[0001] The present application relates to large-scale data processing systems, and more particularly to systems and methods for reducing the reading and writing of data while reducing the number of random access I / O requests to a hard disk drive (HDD) to reduce the overall execution time during complex data processing operations. Background Art

[0002] Spark TM Hadoop TM 、TensorFlow TM and Dryad TM Large-scale data analysis systems such as Riffle are often used to perform complex data analysis operations. Such systems store data partitions in memory for pipeline operators and store data on hard drives across stages with wide dependencies for fault tolerance. In such systems, when running many small tasks into which each job is divided for multi-stage data analysis, all-to-all data transfers - the so-called shuffle operations - become a scaling bottleneck. Zhang et al., "Riffle: Optimized ShuffleReduce Service for Large-Scale Data Analytics", ACM, Proceedings of the 13th EuroSys Conference, page 43, April 23-26, 2018, commented that this bottleneck is due to the superlinear increase in disk I / O operations as the amount of data increases. To address the increase in disk I / O operations, Zhang et al. suggested effectively merging fragmented intermediate shuffle files into large block files to convert small random disk I / O requests into large sequential requests.

[0003] Unfortunately, such solutions come with considerable costs. For example, Riffle TM The system needs to perform two reads and writes for all data, which results in a large I / O overhead. It is still desirable to establish additional methods to reduce the reading and writing of data while reducing the number of random access I / O requests to the hard disk drive (HDD) to further reduce the total execution time during complex data processing operations such as data analysis. Summary of the invention

[0004] Various examples are now described to introduce a group of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] In a sample embodiment, a Shuffle Reduce task is designed and integrated into a data analysis framework. A driver module in the data analysis framework decides whether to use the Shuffle Reduce task and how many Shuffle Reduce tasks to use. The Shuffle Reduce task described in this article performs sequential read I / O requests instead of a large number of random access I / O requests performed in traditional methods (reducing I / O overhead). TM Compared to the 3D image processing system, the size of data read by the shuffle-reduce task described in this paper is significantly reduced, resulting in less I / O overhead and improved overall execution time.

[0006] In an example embodiment, the shuffle-reduce module can be included in the data analysis framework driver, which decides whether to call the shuffle-reduce task for the job (how many to call, etc.) or maintain the existing execution plan. If the shuffle-reduce task is called, it is called as part of the job execution plan. This solution can be applied to any data flow execution platform.

[0007] According to a first aspect of the present disclosure, a method for performing a shuffle-reduce operation is provided, which groups and concatenates data between map and reduce stages during a data analysis process for data conversion. The method includes receiving at least two input files from a first memory as input, wherein each input file has been classified and written by a different mapping task; obtaining batch data from each input file; merging and classifying the batch data in a second memory to form a unified data segment. A shuffle-reduce operation is applied to the unified data segment to generate output data. In an example embodiment, the shuffle-reduce operation includes a commutative reduction operation that provides an output data amount that is less than the input data amount. The output data is written to a third memory, and these steps are repeated until the data from each input file is completely consumed and the analysis data output of the data analysis process has been fully formed.

[0008] According to a second aspect of the present disclosure, a data analysis system is provided, comprising: at least one processor; a first memory storing at least two input files, each input file being classified and written by a different mapping task of a data analysis process; a second memory; a third memory; and an instruction memory storing instructions, wherein the instructions, when executed by the at least one processor, perform a data analysis process, wherein the data analysis process includes a shuffle-reduce operation for grouping and concatenating data between mapping and reduction stages during the data analysis process to perform data transformation. In an example embodiment, the shuffle-reduce operation includes: (1) receiving at least two input files from the first memory as input; (2) obtaining batch data from each input file; (3) merging and classifying the batch data in the second memory to form a unified data segment; (4) applying the shuffle-reduce operation to the unified data segment to generate output data, wherein the shuffle-reduce operation includes a commutative reduction operation that provides an output data amount that is less than the input data amount; (5) writing the output data to the third memory; and (6) repeating (2)-(5) until the data from each input file is completely consumed and the analysis data output of the data analysis process has been completely formed.

[0009] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising instructions, which, when executed by at least one processor, cause the processor to perform a shuffle-reduce operation, wherein the shuffle-reduce operation groups and concatenates data between map and reduce stages during a data analysis process to perform data conversion by performing operations, the operations comprising: (1) receiving at least two input files from a first memory as input, wherein each input file has been classified and written by a different map task; (2) obtaining batch data from each input file; (3) merging and classifying the batch data in a second memory to form a unified data segment; (4) applying a shuffle-reduce operation to the unified data segment to generate output data, wherein the shuffle-reduce operation comprises a commutative reduction operation that provides an output data amount that is less than the input data amount; (5) writing the output data to a third memory; and (6) repeating (2)-(5) until the data from each input file is completely consumed and the analysis data output of the data analysis process has been completely formed.

[0010] In a first implementation of any of the foregoing aspects, the first memory and the third memory include hard disk drives, and the second memory includes dynamic random access memory (DRAM).

[0011] In a second implementation of any of the foregoing aspects, the first memory may include at least one hard disk drive, and the third memory may include at least one of a solid state disk and a persistent memory.

[0012] In a third implementation of any of the foregoing aspects, obtaining batches of data from each input file includes obtaining a total amount of data that does not exceed a memory capacity allocated for the shuffle-reduce operation.

[0013] In a fourth implementation of any of the aforementioned aspects, the driver module determines whether to perform the shuffle-reduce operation on a specific job in the data analysis process based on: whether the job includes a shuffle operation, whether the workload of the job is large enough to obtain performance gain, whether the job includes the commutative reduction operation that provides an output data volume that is smaller than the input data volume in an output file, how many tasks to start to implement the shuffle-reduce operation, and when to start and stop the shuffle-reduce operation.

[0014] In a fifth implementation of any of the aforementioned aspects, the data analysis process is implemented on a data analysis platform, and applying the shuffle-reduce operation includes applying at least one of a key aggregation operation, a key grouping operation, and a key reduction operation as the commutative reduction operation.

[0015] In a sixth implementation of any of the foregoing aspects, the first task performs the steps of receiving input files, obtaining batch data from each input file, and merging and classifying the batch data, and the second task performs the steps of applying a shuffle-reduce operation, and writing output. In an example embodiment, the first task and the second task communicate directly with each other independently of the first memory and the third memory.

[0016] In a seventh implementation of any preceding aspect, applying the shuffle-reduce operation to the unified data segment to generate output data includes applying a plurality of tasks to perform the shuffle-reduce operation.

[0017] The method can be executed and the instruction on the computer-readable medium can be processed by the device, and the other features of the method and the instruction on the computer-readable medium are produced by the functionality of the device. In addition, the explanation provided for each aspect and its implementation is equally applicable to other aspects and corresponding implementations. Different embodiments can be implemented with hardware, software or any combination thereof. In addition, any one of the aforementioned examples can be combined with any one or more of the other aforementioned examples to create new embodiments within the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings illustrate generally, by way of example and not limitation, various embodiments discussed in this document.

[0019] Figure 1 Spark TMA logical view of a job that applies transformations (maps and filters) to data from two separate tables, joining and aggregating items per key (a field of the item) using a GroupByKey function.

[0020] Figure 2 Shows Figure 1 The Spark job shown in TM Execute the plan.

[0021] Figure 3 The mapping between the map and reduce tasks during the shuffle phase is shown.

[0022] Figure 4 Shows Riffle TM A data analysis system, which requires less read and write I / O overhead due to the reduced number of tasks, but needs to run two read and write operations on the entire shuffled data.

[0023] Figure 5 A sample embodiment of a shuffle-reduce method for reducing I / O overhead is shown.

[0024] Figure 6 The steps performed by a shuffle-reduce operation in a sample embodiment are shown.

[0025] Figure 7 A flow chart of a shuffle-reduce operation in a sample embodiment is shown.

[0026] Figure 8 FIG. 4 shows an example of performing a shuffle-reduce operation on a specified input file in a sample embodiment.

[0027] Fig. 9 is a block diagram illustrating a circuit for performing a method according to an example embodiment. DETAILED DESCRIPTION

[0028] First, it should be understood that although the following provides an exemplary implementation of one or more embodiments, Figures 1 to 9 The disclosed systems and / or methods described may be implemented using any number of techniques, whether currently known or existing. The present disclosure should in no way be limited to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0029] In one embodiment, the functions or algorithms described herein can be implemented in software. The software may include computer executable instructions stored on a computer-readable medium or computer-readable storage device such as one or more non-transient memories or other types of local or network-based hardware storage devices. In addition, such functions correspond to modules, which can be software, hardware, firmware, or any combination thereof. Multiple functions can be run in one or more modules as desired, and the described embodiments are only examples. The software can be executed on a digital signal processor, an ASIC, a microprocessor, or other types of processors operating on a computer system such as a personal computer, a server, or other computer system to turn such a computer system into a specially programmed machine.

[0030] Spark TM Hadoop TM 、TensorFlow TM and Dryad TM Large data analysis platforms such as usually use a directed acyclic graph (DAG) model to define user jobs. For example, Figure 1 Spark TM Logical view of a job that applies a map transformation 110 and a filter transformation 120 to input data 130 and a map transformation 140 to input data 150, where the input data 130 and 150 come from two separate tables. The items on each key (a field of the item) are joined and aggregated using, for example, a join function 160. After filtering by a filter task 170, the output data 180 is stored in a result table.

[0031] Figure 2 Shows Figure 1 The Spark job shown in TM Execution plan. As shown in the figure, the DAG execution platform (such as Spark) decides how to map these operations to physical resources. Since operations may occur across partitions, cross-partition communication is required for processing. For narrow dependencies (map and filter operations 210 and 220), Spark TM The system pipelines the transformations and executes the operators in a single state on each partition 230 and 240. Internally, Spark TMThe system attempts to keep the intermediate data of a single task in memory during the map phase so that pipeline operators can be executed efficiently (for example, the filter operator 220 follows the map operator 210 in the first phase). However, operations like map 210, filter 220, etc. result in narrow dependencies between tasks (a one-to-one communication pattern). Typically, these operations are bundled together and executed by the same process. On the other hand, operations like join 250, reduce by key (ReduceByKey), group by key (GroupByKey), etc. result in wide dependencies between different tasks in the reduction phase before filtering by the filter task 260 (a many-to-many communication pattern). Traditionally, the data exchange between the map phase and the reduce phase is represented as a shuffle period, such as Figure 2 shown.

[0032] As used herein, the shuffle phase includes all-to-all communication for wide dependencies between phase 1 (map phase) and phase 2 (reduce phase). Each map task 210 reads from a data partition (e.g., several rows of a large table), converts the data into an intermediate format using a map task operator, classifies and aggregates items through the partitioning function (e.g., key range) of the reduce phase to produce item blocks, and saves the blocks to intermediate files on disk. The map task 210 also writes a separate index file that shows the offsets of the blocks corresponding to each reduce task of the reduce phase. Each reduce task brings the specified data blocks together and performs the reduce task operation. By looking up the offset in the index file, each reduce task issues a get request for the target block from all map output files. Therefore, data that was originally partitioned according to each table row is processed and shuffled to data partitioned according to the reduction key range. Between phases with wide dependencies, each reduce task needs to read data blocks from all map task outputs. If the intermediate shuffle file is not permanent, even a single reduce task failure may cause the entire map phase to be recalculated. Therefore, for strong fault tolerance, it is very important to persist the shuffle data.

[0033] Therefore, the shuffle operation in the shuffle phase is an extremely resource-intensive operation. Each data block transferred from the map task to the reduce task requires data serialization, disk and network I / O, and data deserialization. Despite this, the shuffle operation is used extensively in various types of jobs. For example, jobs that need to partition, group, or reduce or concatenate data by key all involve shuffle operations. The shuffle phase can be used for operations such as determining how many times a specific word appears in a text, where the individual results of each map task are combined to obtain the final result.

[0034] Previous schemes let map tasks write their output data to permanent storage for fault tolerance (if a reduce task fails, there is no need to rerun any map tasks). For large workloads, hard disk drives (HDDs) are preferred over other types of permanent storage because they are cheap and easy to scale. However, this initial design does not scale well as the number of map and reduce tasks increases. The reduce tasks need to perform M·R random access input / output (I / O) requests to the HDD, where M is the number of map tasks and R is the number of reduce tasks. Therefore, the shuffle period causes a lot of I / O overhead.

[0035] Figure 3 This approach is summarized in FIG, which shows the mapping between map tasks 310 and corresponding reduce tasks 360, 370, and 380 during the shuffle phase. Figure 3 As shown, each map task 310 generates a corresponding data set 330, 340, and 350, which are directed to corresponding reduce tasks 360, 370, and 380 as shown. As shown, data set 330 is processed by reduce task 360, data set 340 is processed by reduce task 370, and data set 350 is processed by reduce task 380.

[0036] As the job data size increases, the number of map and reduce tasks also grows proportionally. Because each reduce task R needs to fetch from all map tasks M, the number of shuffle I / O requests M·R increases quadratically, and the average block size per fetch S / (M·R) decreases quadratically. Using fewer reduce tasks R reduces the total number of shuffle fetches, thus improving shuffle performance. However, using fewer tasks inevitably expands the average size of input data and creates very large, slow tasks that must spill intermediate data to permanent storage, thus increasing overhead.

[0037] Therefore, shuffle operations (e.g., joins) are the most difficult operations to perform in a distributed data analytics platform. Shuffle operations for large workloads write data to permanent storage, so that in the event of a reduce task failure, there is no need to rerun the entire map phase. However, as the most cost-effective solution, a large amount of data is temporarily stored in a hard disk drive (HDD). Random access I / O requests cause significant overhead on the HDD.

[0038] The aforementioned Riffle TM The system provides a shuffle service that takes a different approach to address the scaling bottleneck caused by all-to-all data transfers for shuffle operations. TM The system introduces a merge task in the middle of the shuffle period. TMInstead of providing the data written by the map task directly to the reduce task, the system assigns the merge task to read and merge the fragmented intermediate shuffle files produced by the map task into larger block files, thereby converting small, random disk I / O requests into large, sequential requests. As a result, the reduce task performs far fewer random access I / O requests to the HDD. In this case, Riffle TM The system performs M·R / N random access I / O requests, where N is the average number of mapping tasks associated with each merge task.

[0039] However, Riffle TM The performance improvements provided by the system come at a considerable cost. Figure 4 As shown, although Riffle TM The system requires less write and read I / O overhead due to the reduction in the number of tasks (by factor N), but Riffle TM The system performs two reads and writes of the entire shuffle data—one for reading the data sets 330, 340, and 350 created by the map task 310, and one for reading the merged data set 420, 430, and 440 created by the merge task 410. Doubling the reads and writes of the entire shuffle data still results in a large I / O overhead, especially for the larger block files created during the merge operation 410.

[0040] The systems and methods described herein further reduce I / O overhead in a cost-effective manner by implementing a shuffle phase for a data analysis system that essentially avoids reading and writing the entire shuffled data twice (e.g., Riffle TM system), while also reducing the number of random access I / O requests to the HDD (such as Riffle TM system). With Riffle TM Compared with conventional implementations in the system, the system and method provided herein reduces the total execution time of the shuffle period and reduces the number of I / O requests to the HDD by taking advantage of two characteristics that user workloads often exhibit, namely:

[0041] 1. The reduction operation is commutative.

[0042] 2. Reduction operation reduces the amount of data (output data size is much smaller than input data size).

[0043] Based on these two characteristics, a shuffle-reduce operation is provided that combines a merge file operation with a reduce operation. In the exemplary embodiment, the shuffle-reduce operation is a commutative reduce operation implemented during the shuffle period, which provides an output data amount that is smaller than the input data amount. Since the reduce operation is commutative, the final result is correct because the final result is the same as the previous method. In addition, the shuffle-reduce operation eventually issues a Riffle-like signal to the HDD. TM The same number of I / O requests are in the system. However, the second batch of reads and writes is performed on significantly less data because only the data produced by the aggregation operation is stored.

[0044] Figure 5 An example embodiment of a shuffle-reduce method for reducing I / O overhead is shown. As shown in the figure, Figure 4 The merge operation 410 of the mapping task 310 is replaced by a shuffle-reduce task 510, which merges the corresponding data sets 330, 340 and 350 from the corresponding map task 310 and reduces the data sets by performing a reduction operation to produce reduced-size data sets 520, 530 and 540 corresponding to the data sets 330, 340 and 350, respectively. As shown, for fault tolerance, the data sets 330, 340 and 350 and the reduced-size data sets 520, 530 and 540 are stored in the HDD. Then, the reduce tasks 360, 370 and 380 are applied to the reduced-size data sets 520, 530 and 540, respectively. Due to the reduced size of the data sets 520, 530 and 540, the total execution time of the resulting shuffle phase is significantly reduced.

[0045] Figure 6 shows the steps performed by the shuffle-reduce operation, and Figure 7 FIG. 4 shows a flow chart of the shuffle-reduce operation in the example embodiment. Figure 6 and Figure 7 As shown, by receiving a plurality of input files 610 ( Figure 7 In the example embodiment, each input file 610 is written by a different mapping task in a different partition. Then, in step (2), small batches of data 620 are obtained from all input files 610, such as Figure 6 As shown by arrow 630 in ( Figure 7 In the example embodiment, the total size of the acquired data should not exceed the DRAM capacity of the shuffle-reduce task. Then, in step (3), the small batches of data are merged and sorted together (according to their keys) to form unified data 640 ( Figure 7 Operation 730 in FIG.

[0046] In step (4), a shuffle-reduce operation is applied to the unified data 640 to produce a reduced data set 650 ( Figure 7 In an example embodiment, the shuffle-reduce operation is applied when the reduction operation is commutative and reduces the amount of data so that the output data size is much smaller than the input data size. For example, in Spark TM In the system, such reduction operations include key aggregation operations, key reduction operations, and key grouping operations. TM AggregateByKey operations in the system group values ​​relative to a common key using a given combining function and a neutral "zero value" to return values ​​of different types for the key. For example, the AggregateByKey operation allows a tuple of a student, subject, and multiple scores to be processed into an output dataset consisting of the student and the highest score or percentage for a subject. Spark TM The ReduceByKey operation in the system merges the values ​​for each key using an associative reduction function that takes two arguments and returns a single element and is mathematically commutative and associative. In other words, when repeatedly applied to the same dataset with multiple partitions, the ReduceByKey function produces the same results regardless of the order of the elements. Spark TM The GroupByKey function in the system is very similar to the ReduceByKey function. The GroupByKey function takes a key-value pair (K, V) as input and collects the values ​​of each key in the form of an iterator to produce an output including a list of keys and values. Therefore, the GroupByKey function groups all values ​​with respect to a single key and returns them in the form of an iterator. Multiple pipeline operations that reduce the amount of data can also be used for shuffle-reduce operations. Similar functions of other data analysis systems will be obvious to those skilled in the art.

[0047] Then, in step (5), the reduced data set 650 is written to the external HDD, as indicated by arrow 660 ( Figure 7 When there is more data to be processed from input file 610, steps (2)-(5) are repeated at step (6) until all input data has been processed ( Figure 7 When there is no more input data to be processed, the process ends at operation 770.

[0048] Therefore, the shuffle-reduce operation reduces the size of the data read by the reduction task (compare the data size after step (6) with Figure 6 This significantly reduces the I / O overhead and reduces the total execution time of the shuffle phase. Due to the reduced data size, HDDs can be used in a cost-effective solution for fault tolerance.

[0049] In addition to the shuffle-reduce tasks described herein, the sample embodiments may include a shuffle-reduce module included in a data analysis framework driver. The data analysis framework driver is the master node in the data analysis application, which splits the data analysis application into tasks and arranges these tasks to run on executors. The driver module can generate tasks across multiple partitions. In the sample embodiment, a driver module is provided on the client side to decide whether to call the shuffle-reduce tasks described herein for the job (how many to call, etc.) or to maintain the existing execution plan. In the sample embodiment, the driver module of the traditional data analysis framework driver is modified to determine when to call the shuffle-reduce task by checking the following:

[0050] 1. Is there a shuffling period?

[0051] 2. Whether the workload is large enough for the shuffle-reduce approach to gain benefits; and

[0052] 3. Is it known (or determined) that the reduction operation is commutative and will reduce the amount of data (the output data size is much smaller than the input data size).

[0053] When these conditions are met, the shuffle-reduce task described in this article is called.

[0054] A first alternative to the above embodiment divides the shuffle-reduce phase into two different tasks (merge, reduce). This embodiment will have similar features to the above embodiment. All operations before the merge operation will be performed by the merge task, and the rest will be performed by the reduce task. The merge task will communicate directly with the reduce task, rather than through the HDD.

[0055] The benefit of the first alternative is that the data analysis framework user / operator can independently scale up / down the merge and reduce operations in the shuffle phase, which provides greater flexibility and better resource utilization. The disadvantage is that this embodiment adds an extra layer of network communication because it gets rid of the locality aspect of the first embodiment.

[0056] A second alternative to the above embodiment would be to use a different storage medium instead of the HDD. For cost purposes, HDDs are generally preferred for large amounts of data because they are cheaper than the alternatives. However, since the shuffle-reduce task significantly reduces the amount of data processed by the reduce task, one possible solution is to store the data in a more efficient medium (such as a solid-state disk (SSD) or a different persistent memory technology) in order to obtain better performance for the shuffle-reduce phase (since the cost of these solutions is now reduced due to storing less data). Therefore, the second alternative provides better I / O performance at the expense of the overall cost of the entire architecture.

[0057] Figure 8An example of a shuffle-reduce operation is shown for a specified input file in a sample embodiment. In this example, several input files 810 including different types of fruits (apple, blueberry, grape, melon, orange, mango) are merged at a merge operation 820 to generate corresponding files 830 and 840 stored in DRAM. A reduction task 850 performs a ReduceByKey (total) operation on the corresponding files 830 and 840 to generate corresponding output files 860 and 870. It will be appreciated by those skilled in the art that ReduceByKey is a commutative operation, and the resulting files 860 and 870 are smaller than files 830 and 840. As shown, the output key-value pairs include the fruit and the number of occurrences in the merged input files 830 and 840. In this example, by combining repeated elements into element and element count key-value pairs, the number of key-value pairs has been reduced from 12 to 8, thereby saving memory space. It should be understood that similar reductions in memory space can be achieved for other commutative reduction operations such as AggregateByKey operations or GroupByKey operations.

[0058] The shuffle-reduce phase reduces HDDI / O overhead because it significantly reduces the number of random access I / O requests. The shuffle-reduce task described in this paper also significantly reduces the size of data that needs to be read by the reduce task. As a result, the I / O overhead and the total execution time of the shuffle phase are greatly reduced.

[0059] The following table compares Spark TM System, Riffle TM Performance of the system's traditional data analysis framework and the shuffle-reduce approach implemented in this paper.

[0060]

[0061] in:

[0062] M is the number of mapping tasks;

[0063] R is the number of reduction tasks;

[0064] N is the number of map tasks per merge or shuffle-reduce task;

[0065] S is the total shuffled data size; and

[0066] R is a reduction factor (0 < r ≤ 1, depending on workload, operations, and N). In the sample embodiment, r < 0.1, but varies by workload. In the worst case, r = 1 and the total read / write size is still at least the same as Riffle TM The read / write size of the system is equally good.

[0067] Fig. 9A general purpose computer 900 is shown that is suitable for implementing one or more embodiments of the methods disclosed herein. For example, Fig. 9 The computer 900 in the embodiment may be used in a data analysis framework of the type described herein (e.g., Spark TM Hadoop TM 、TensorFlow TM and Dryad TM ) is implemented on a computer 900. The above components may be implemented on any general-purpose network component, such as a computer 900 that has sufficient processing power, memory resources, and network throughput capabilities to handle the necessary workload placed on it. The computer 900 includes a processor 910 (which may be referred to as a central processor unit or CPU) that communicates with memory devices, including auxiliary storage 920, read-only memory (ROM) 930, random access memory (RAM) 940, input / output (I / O) devices 950, and network connection devices 960. In an example embodiment, the network connection device 960 further connects the processor 910 to a client-side data analysis driver 970, which manages the data analysis process and determines when to call the shuffle-reduce task as described herein. The processor 910 may be implemented as one or more CPU chips, or may be part of one or more application-specific integrated circuits (ASICs).

[0068] The auxiliary storage 920 is usually composed of one or more disk drives or tape drives, and is used for non-volatile storage of data, and is used as an overflow data storage device if the RAM 940 is not large enough to accommodate all working data. The auxiliary storage 920 can be used to store the program loaded into the RAM 940 when the program is selected for execution. The ROM 930 is used to store instructions and possible data read during program execution. The ROM 930 is a non-volatile memory device that usually has a small memory capacity relative to the larger memory capacity of the auxiliary storage 920. The RAM 940 is used to store volatile data and may be used to store instructions. Access to both the ROM 930 and the RAM 940 is usually faster than access to the auxiliary storage 920.

[0069] It should be understood that the computer 900 can execute instructions from a computer-readable non-transitory medium storing computer-readable instructions and one or more processors coupled to the memory, and when executing the computer-readable instructions, the computer 900 is configured to perform the functions referred to in the present disclosure. Figures 1 to 8 The method steps and operations described herein. Computer-readable non-transitory media include all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media.

[0070] It should also be understood that software including one or more computer executable instructions that facilitate processing and operations as described above with reference to any or all of the steps of the present disclosure can be installed on and sold with one or more servers or databases. Alternatively, the software can be acquired and loaded into one or more servers or one or more databases in a manner consistent with the present disclosure, including acquiring the software through a physical medium or distribution system, including, for example, from a server owned by the creator of the software or from a server that the creator of the software does not own but uses. For example, the software can be stored on a server for distribution over the Internet.

[0071] In addition, it will be appreciated by those skilled in the art that the present disclosure is not limited to the details of the constructions and the arrangement of the parts to which it is applied, which are described in the following description or shown in the accompanying drawings. The embodiments herein may be other embodiments and can be practiced or executed in various ways. In addition, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. "Including," "comprising," or "having" and their variations used herein are intended to cover the items listed thereafter and their equivalents and additional items.

[0072] The components of the exemplary devices, systems, and methods employed in accordance with the illustrated embodiments may be implemented at least in part in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. For example, these components may also be implemented as a computer program product such as a computer program, program code, or computer instructions tangibly embodied in an information carrier or machine-readable storage device for execution by a data processing device, such as a programmable processor, a computer, or multiple computers, or to control the operation of the data processing device.

[0073] The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to execute on a single computer or on multiple computers at a site or distributed on multiple sites and interconnected by a communication network. In addition, the functional programs, codes, and code segments used to complete the systems and methods described herein can be easily interpreted as within the scope of the present disclosure by programmers with the technology of the field to which the present disclosure belongs. The method steps associated with the exemplary embodiments can be executed by one or more programmable processors to execute computer programs, codes, or instructions to perform functions (e.g., by operating on input data and generating output). For example, the method steps can also be performed by the following and the device can be implemented as follows: a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0074] The various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration.

[0075] Processors suitable for executing computer programs include, for example, both general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or from both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, or one or more mass storage devices operably coupled to store data to receive data from or transmit data to, or receive data from and transmit data to, the one or more mass storage devices such as magnetic disks, magneto-optical disks, or optical disks. Information carriers suitable for containing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks or removable disks, magneto-optical disks, CD-ROM disks, or DVD-ROM disks). The processor and memory can be supplemented by or included in a dedicated logic circuit.

[0076] Those skilled in the art will appreciate that any of a variety of different techniques and technologies may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0077] Those skilled in the art will also appreciate that the various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been described above generally according to their functions. Whether this function is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. The software module can reside in a random access memory (RAM), a flash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art. The sample storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor. In other words, the processor and the storage medium can reside in an integrated circuit or be implemented as discrete components.

[0078] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read-only memory (EEPROM)), and any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media that can store instructions for execution by one or more processors, so that the instructions, when executed by one or more processors, cause the one or more processors to perform any one or more methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or devices. As used herein, the term "machine-readable medium" does not include the signal itself.

[0079] Although several embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. Other steps may be provided from the described flows, or steps may be eliminated, and other components may be added to or removed from the described systems. Other embodiments may be within the scope of the appended claims.

Claims

1. A method of performing a shuffle-reduce operation that groups and concatenates data between map and reduce phases to perform data transformation during a data analysis process, the method comprising: (1) receiving at least two input files from a first memory as input, wherein each input file has been classified and written by a different mapping task; (2) obtaining batch data from each input file; (3) merging and classifying the batch data in the second memory to form a unified data segment; (4) applying a shuffle-reduce operation to the unified data segment to generate output data, wherein the shuffle-reduce operation includes a commutative reduction operation, and the commutative reduction operation provides an output data amount that is less than the input data amount; (5) writing the output data into a third memory; as well as (6) repeating (2)-(5) until the data from each input file is completely consumed and the analysis data output of the data analysis process is completely formed; Wherein, the data analysis process is implemented on a data analysis platform, and applying the shuffle-reduce operation includes applying at least one of a key aggregation operation, a key grouping operation and a key reduction operation as the commutative reduction operation, wherein the key aggregation operation uses a given combination function and a neutral "zero value" to group values ​​relative to a common key to return different types of values ​​for the key, the key reduction operation uses an associative reduction function to merge each key-value pair, and the key grouping operation takes the key-value pair as input and collects the value of each key in the form of an iterator to produce an output including a list of the key and the value.

2. The method according to claim 1, wherein: The first memory and the third memory include hard disk drives, and the second memory includes a dynamic random access memory DRAM.

3. The method according to claim 1, wherein: The first memory includes at least one hard disk drive, and the third memory includes at least one of a solid state drive and a persistent memory.

4. The method according to claim 1, wherein: Retrieving batches of data from each input file includes retrieving a total amount of data that does not exceed a memory capacity allocated for the shuffle-reduce operation.

5. The method of claim 1 , further comprising a driver module that determines whether to perform the shuffle-reduce operation on a specific job during the data analysis process based on whether the job includes a shuffle operation, whether the workload of the job is large enough to obtain a performance gain, whether the job includes the commutative reduce operation that provides an output data volume in an output file that is smaller than an input data volume, how many tasks must be started to implement the shuffle-reduce operation, and when the shuffle-reduce operation will start and stop.

6. The method according to claim 1, further comprising performing steps (1)-(3) by a first task and performing steps (4)-(5) by a second task, wherein: The first task and the second task communicate directly with each other independently of the first memory and the third memory.

7. The method according to claim 1, wherein: Applying the shuffle-reduce operation to the unified data segment to generate output data includes applying a plurality of tasks to perform the shuffle-reduce operation.

8. A data analysis system comprising: at least one processor; A first memory storing at least two input files, each input file being classified and written by a different mapping task of a data analysis process; A second memory; A third memory; and an instruction memory storing instructions that, when executed by the at least one processor, perform a data analysis process, the data analysis process including a shuffle-reduce operation for grouping and concatenating data between map and reduce stages to perform data transformation during the data analysis process, the shuffle-reduce operation comprising: (1) receiving the at least two input files from the first memory as input; (2) obtaining batch data from each input file; (3) merging and classifying the batch data in the second memory to form a unified data segment; (4) applying a shuffle-reduce operation to the unified data segment to generate output data, wherein the shuffle-reduce operation includes a commutative reduction operation, and the commutative reduction operation provides an output data amount that is less than the input data amount; (5) writing the output data into the third memory; and (6) repeating (2)-(5) until the data from each input file has been completely consumed and the analysis data output of the data analysis process has been completely formed; The system also includes a data analysis platform, which implements the data analysis process and applies the shuffle reduction operation by applying at least one of a key aggregation operation, a key grouping operation and a key reduction operation as the commutative reduction operation, wherein the key aggregation operation uses a given combination function and a neutral "zero value" to group values ​​relative to a common key to return different types of values ​​for the key, the key reduction operation uses an associative reduction function to merge each key-value pair, and the key grouping operation takes the key-value pair as input and collects the values ​​of each key in the form of an iterator to produce an output including a list of the key and the value.

9. The system according to claim 8, wherein: The first memory and the third memory include hard disk drives, and the second memory includes a dynamic random access memory DRAM.

10. The system according to claim 8, wherein: The first memory includes at least one hard disk drive, and the third memory includes at least one of a solid state drive and a persistent memory.

11. The system according to claim 8, wherein: The total amount of data retrieved from each input file does not exceed the memory capacity allocated by the data analysis process for the shuffle-reduce operation.

12. The system of claim 8, further comprising a driver module that determines whether to perform the shuffle-reduce operation on a specific job in the data analysis process based on whether the job includes a shuffle operation, whether the workload of the job is large enough to obtain a performance gain, whether the job includes the commutative reduce operation that provides an output data volume in an output file that is smaller than an input data volume, how many tasks must be started to implement the shuffle-reduce operation, and when the shuffle-reduce operation will start and stop.

13. The system according to claim 8, further comprising a first task for executing steps (1)-(3) and a second task for executing steps (4)-(5), wherein: The first task and the second task communicate directly with each other independently of the first memory and the third memory.

14. The system according to claim 8, wherein: The shuffle-reduce operation includes a plurality of tasks that are applied to the unified data segment to generate output data.

15. A computer-readable storage medium comprising instructions that, when executed by at least one processor, cause the processor to perform a shuffle-reduce operation, the shuffle-reduce operation grouping and concatenating data between map and reduce phases to perform data transformation during a data analysis process by performing operations, the operations comprising: (1) receiving at least two input files from a first memory as input, wherein each input file has been classified and written by a different mapping task; (2) obtaining batch data from each input file; (3) merging and classifying the batch data in the second memory to form a unified data segment; (4) applying a shuffle-reduce operation to the unified data segment to generate output data, wherein the shuffle-reduce operation includes a commutative reduction operation, and the commutative reduction operation provides an output data amount that is less than the input data amount; (5) writing the output data into a third memory; as well as (6) repeating (2)-(5) until the data from each input file has been completely consumed and the analysis data output of the data analysis process has been completely formed; Wherein, the data analysis process is implemented on a data analysis platform, and the medium also includes instructions, which, when executed by the at least one processor, cause the processor to perform operations, wherein the operations include applying at least one of a key aggregation operation, a key grouping operation, and a key reduction operation as the commutative reduction operation, wherein the key aggregation operation uses a given combination function and a neutral "zero value" to group values ​​relative to a common key to return different types of values ​​for the key, the key reduction operation uses an associative reduction function to merge each key-value pair, and the key grouping operation takes the key-value pair as input and collects the values ​​of each key in the form of an iterator to produce an output including a list of the key and the value.

16. The medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the processor to perform steps (1)-(3) via a first task and steps (4)-(5) via a second task, wherein: The first task and the second task communicate directly with each other independently of the first memory and the third memory.

17. The medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the processor to perform a plurality of tasks to implement the shuffle-reduce operation.

Images