Data processing method, device and storage medium

Abstract

The embodiment of the present application provides a data processing method, device and storage medium, the method comprising: obtaining data to be processed, the data to be processed is file data written by a first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in a second executor; merging the data to be processed corresponding to the same Reducer to obtain merged data, wherein the number of merged data is less than the number of files corresponding to the file data; receiving a first read operation sent by the second executor, the first read operation is used to read the merged data corresponding to the Reducer; according to the first read operation, sending the merged data corresponding to the Reducer. The embodiment of the present application can reduce the number of random reads, thereby speeding up the shuffle read process and reducing disk IO resources and network consumption.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information processing technology, and in particular, to a data processing method, device and storage medium. Background Art

[0002] In the MapReduce framework, shuffle is the bridge between Map and Reduce. The output of Map must go through the shuffle process to be used in Reduce. Therefore, the performance of shuffle directly affects the performance and throughput of the entire framework.

[0003] As an implementation of the MapReduce framework, Spark naturally implements the shuffle logic. When Spark runs an application that includes a shuffle process, the executor, in addition to running the task, is also responsible for writing the data generated by the task to the disk after the task is completed, and providing shuffle data to other executors. Specifically, Spark can include a write phase and a read phase, namely: shufflewrite and shuffleread. During the shufflewrite process, the Mapper in the executor is sharded according to the number of Reducers in other executors, and then the sharded data is stored in a special directory of Spark, which is managed and provided by the external component external shuffle service; during the shuffleread process, the Reducer reads the data corresponding to itself in each Mapper, resulting in a large amount of random reads, making the entire shuffleread process very slow, and there are a lot of unnecessary disk IO resources and network consumption. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device and storage medium to speed up the shuffle read process and reduce unnecessary disk IO resources and network consumption.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Obtaining data to be processed, where the data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, where N is an integer greater than 1, and the Reducer is located in the second executor;

[0007] Merging the to-be-processed data corresponding to the same Reducer to obtain merged data, wherein the number of the merged data is less than the number of files corresponding to the file data;

[0008] receiving a first read operation sent by the second executor, where the first read operation is used to read the merged data corresponding to the Reducer;

[0009] According to the first read operation, the merged data corresponding to the Reducer is sent.

[0010] In a possible implementation, the number of the merged data is equal to the number of the Reducers.

[0011] In a possible implementation, each merged data corresponds to a block identifier of a corresponding Reducer.

[0012] In a possible implementation, after merging the to-be-processed data corresponding to the same Reducer to obtain the merged data, the process may further include: updating the to-be-processed data.

[0013] In a possible implementation, the data to be processed is metadata corresponding to at least one application.

[0014] In a possible implementation, each of the applications corresponds to a thread, the life cycle of the thread is the same as the life cycle of the corresponding application, and the thread is used to periodically scan the data structure related to the corresponding application and perform a merge processing operation whenever new file data is generated.

[0015] In a possible implementation, the thread is driven by a timer.

[0016] In a possible implementation, the data processing method may further include:

[0017] If an exception is detected during the merging process, a second read operation sent by the second executor is received, where the second read operation is used to read the pre-merged data corresponding to the Reducer;

[0018] According to the second read operation, the pre-merged data corresponding to the Reducer is sent.

[0019] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0020] An acquisition module, used for acquiring data to be processed, wherein the data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in the second executor;

[0021] A processing module, used for merging the to-be-processed data corresponding to the same Reducer to obtain merged data, wherein the number of the merged data is less than the number of files corresponding to the file data;

[0022] A receiving module, used for receiving a first read operation sent by the second executor, where the first read operation is used for reading the merged data corresponding to the Reducer;

[0023] A sending module is used to send the merged data corresponding to the Reducer according to the first reading operation.

[0024] In a possible implementation, the number of the merged data is equal to the number of the Reducers.

[0025] In a possible implementation, the processing module may also be used to: merge the data to be processed corresponding to the same Reducer to obtain the merged data, and then update the data to be processed.

[0026] In a possible implementation manner, the data processing device may further include:

[0027] Monitoring module, used to monitor whether abnormalities occur during the merging process;

[0028] Correspondingly, the receiving module is further used to receive a second read operation sent by the second executor when the monitoring module detects an abnormality in the merging process, and the second read operation is used to read the pre-merging data corresponding to the Reducer;

[0029] The sending module is further used to send the pre-merged data corresponding to the Reducer according to the second reading operation.

[0030] In a third aspect, an embodiment of the present application provides a data processing device, including: a memory and a processor;

[0031] Wherein, the memory is used to store program instructions;

[0032] The processor is used to call and execute the program instructions stored in the memory. When the processor executes the program instructions stored in the memory, the data processing device is used to execute the method described in any implementation manner of the first aspect above.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in any implementation of the first aspect above.

[0034] The data processing method, device and storage medium provided in the embodiments of the present application obtain data to be processed, which is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in the second executor; the data to be processed corresponding to the same Reducer are merged to obtain merged data, wherein the number of merged data is less than the number of files corresponding to the file data; a first read operation sent by the second executor is received, and the first read operation is used to read the merged data corresponding to the Reducer; according to the first read operation, the merged data corresponding to the Reducer is sent. Since the data to be processed corresponding to the same Reducer are merged, and the number of merged data is less than the number of files corresponding to the file data, the number of random reads can be reduced, thereby speeding up the shuffle read process and reducing disk IO resources and network consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0036] Figure 2 A flowchart of a data processing method provided in one embodiment of the present application;

[0037] Figure 3 A flowchart of a data processing method provided by another embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of a data processing device provided in another embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of a data processing device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0041] It should be understood that the numbers "first" and "second" etc. in the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, and should not constitute any limitation on the embodiments of the present application.

[0042] The "and / or" involved in the embodiments of the present application describes the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0043] First, the application scenarios and some vocabulary involved in the embodiments of the present application are introduced.

[0044] Kubernetes is a container cluster orchestration and management system open sourced by Google.

[0045] Spark is a new generation of distributed memory computing framework and a top Apache open source project. Compared with the Hadoop Map-Reduce computing framework, Spark keeps the intermediate computing results in memory, which is 10 to 100 times faster. Shuffle is a specific phase in the MapReduce framework, between the Map phase and the Reduce phase. When the output of Map is to be used by Reduce, the output needs to be hashed by key value and distributed to each Reducer. This process is called shuffle.

[0046] ESS, a peripheral component of Spark, can register the data location written during the shuffle write process to the component and provide shuffle read service.

[0047] Application. In Spark, an application corresponds to a user job.

[0048] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the application scenario of the embodiment of the present application may include but is not limited to: actuator 1, actuator 2, ESS 1, ESS 2 and actuator 3. Of course, the application scenario of the embodiment of the present application does not limit the number of actuators and ESSs, and this embodiment is only described by taking 3 actuators and two ESSs as an example.

[0049] Among them, executor 1 executes Task 0 and Task 1, that is, the Map Task Set (MapTaskSet) executed by executor 1 includes Task 0 and Task 1; executor 2 executes Task 2 and Task 3, that is, the Map Task Set executed by executor 2 includes Task 2 and Task 3; executor 1 and executor 2 are independent of each other and do not interfere with each other. In addition, executor 1 and executor 2 are also responsible for writing shuffle data, that is, writing the data generated by executing tasks into ESS, among which executor 1 writes the data generated by executing Task 0 and Task 1 into ESS 1, and executor 2 writes the data generated by executing Task 2 and Task 3 into ESS 2. ESS1 and ESS 2 are independent of each other and do not interfere with each other.

[0050] Executor 3 executes Task 4, Task 5 and Task 6 to read shuffle data, that is, the Reduce Task Set executed by Executor 3 includes Task 4, Task 5 and Task 6. Specifically, Executor 3 can read the data required to execute Task 4, Task 5 and Task 6 from ESS 1 and ESS 2 respectively.

[0051] In the above application scenario, the end that writes shuffle data is called the Map end, and each task that generates data on the Map end is called a Mapper, for example, Task 0, Task 1, Task 2, and Task 3 are all called Mappers. Correspondingly, the end that reads shuffle data is called the Reduce end, and each task that reads data on the Reduce end is called a Reducer, for example, Task 4, Task 5, and Task 6 are all called Reducers.

[0052] In the following embodiments, the executor that writes shuffle data is referred to as a first executor, the executor that reads shuffle data is referred to as a second executor, and the shuffle data written in the ESS is referred to as data to be processed.

[0053] For the prior art, for example, when Spark runs an application with a shuffle process, the application corresponds to M Mappers and R Reducers, then the entire shuffle read process will generate M×R random reads. In actual applications, M and R are very large, and the number of random reads generated will increase exponentially, resulting in huge consumption of disk IO resources and a long duration of the shuffle read process.

[0054] Based on the above problems, in the embodiment of the present application, in ESS, the to-be-processed data written into ESS through the shuffle write process is merged and processed, thereby reducing the consumption of disk IO resources and the performance loss caused by sending small packets over the network, and improving the operation efficiency.

[0055] The following specific embodiments are used to illustrate the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0056] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application. The present application provides a data processing method, which can be executed by a data processing device, and the data processing device can be implemented in software and / or hardware. Exemplarily, the data processing device can be specifically an ESS, or the data processing device can be built into the ESS, such as a module or chip in the ESS.

[0057] like Figure 2 As shown, the method of the embodiment of the present application includes:

[0058] S201, obtaining data to be processed.

[0059] The data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in the second executor.

[0060] Still refer to Figure 1 Taking the application scenario shown as an example, ESS 1 receives file data P0, file data P1 and file data P2 written by the first executor (i.e. executor 1), that is, obtains the data to be processed. It should be noted that for the same Mapper, the number of files corresponding to the file data is equal to the number of Reducers in the second executor (i.e. executor 3). If ESS 1 corresponds to N Mappers, the number of files corresponding to the file data received in ESS 1 is N times the number of Reducers.

[0061] Considering that in actual applications, the space occupied by the data generated by all Mappers corresponding to ESS may be larger than the memory of ESS, in this case, file data P0, file data P1 and file data P2 are first written to ESS, and then the remaining part is written to ESS. For example, if the remaining part is represented by file data P01, file data P11 and file data P21, then file data P01, file data P11 and file data P21 are written to ESS. The number of files corresponding to the file data contained in the remaining part is also a natural multiple of the number of Reducers.

[0062] S202: Merge the to-be-processed data corresponding to the same Reducer to obtain merged data.

[0063] Among them, the number of merged data is less than the number of files corresponding to the file data. Since the second executor's reading of shuffle data in the ESS may occur during the merge process or after the merge process is completed, the merged data in this step can be the data after the merge process is completed (all the data after the merge process) or the data during the merge process (part of the data after the merge process and part of the data before the merge process). Therefore, the number of merged data is less than the number of files corresponding to the file data, but the number of merged data is greater than or equal to the number of Reducers.

[0064] For example, if the merging process is successfully performed, M×R files will eventually be merged into R files.

[0065] S203: Receive a first read operation sent by the second executor.

[0066] The first read operation is used to read the merged data corresponding to the Reducer.

[0067] It should be noted that when reading data, multi-threaded transmission of the merged data is required, otherwise the bandwidth will not be fully utilized, resulting in a waste of resources; and, if the transmission time of a single file is too long, it will also cause a significant decline in data processing performance.

[0068] S204. According to the first read operation, send the merged data corresponding to the Reducer.

[0069] In this embodiment, after obtaining the data to be processed, the data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in the second executor, and the data to be processed corresponding to the same Reducer is merged to obtain merged data, wherein the number of merged data is less than the number of files corresponding to the file data, and after receiving the first read operation sent by the second executor, the first read operation is used to read the merged data corresponding to the Reducer, and according to the first read operation, the merged data corresponding to the Reducer is sent. Since the data to be processed corresponding to the same Reducer is merged, and the number of merged data is less than the number of files corresponding to the file data, the number of random reads can be reduced, thereby speeding up the shuffle read process and reducing disk IO resources and network consumption.

[0070] Optionally, the number of merged data is equal to the number of Reducers. Those skilled in the art will understand that when the merge process is completed normally, the number of merged data is equal to the number of Reducers; if an exception occurs in the merge process, part of the data to be processed is merged, and the remaining part of the data to be processed remains unchanged. At this time, the number of merged data is greater than the number of Reducers.

[0071] After the merging process is completed normally, in some embodiments, S202, after merging the to-be-processed data corresponding to the same Reducer and obtaining the merged data, the above data processing method may further include: updating the to-be-processed data. That is, during the merging process, the to-be-processed data needs to be kept unchanged and no update operation is performed on it. Only after the merging process is completed, the to-be-processed data that has completed the merging process is updated to prevent abnormal phenomena from causing data inconsistency.

[0072] Optionally, each merged data corresponds to a block identifier of a corresponding Reducer. In this way, the Reducer can map the corresponding merged data by the block identifier, that is, change the shuffle read interface to adapt to the structure of the ESS provided in the embodiment of the present application.

[0073] For example, refer to Figure 1 In the application scenario shown, when executor 3 (i.e., the second executor) reads shuffle data, it needs to map according to the transmitted block identifier (BlockId). The original shuffle read interface is Seq (Block_id1…Block_idn), which should be mapped to the merged BlockId.

[0074] Furthermore, the data to be processed may be metadata corresponding to at least one application, wherein the processing of metadata corresponding to different applications is independent of each other and does not interfere with each other.

[0075] Optionally, each application corresponds to a thread, and the life cycle of the thread is the same as the life cycle of the corresponding application. That is, when the application is registered, the thread corresponding to the application is generated, and when the application is executed, the thread corresponding to the application is destroyed. The thread is used to periodically scan the data structure related to the corresponding application, and perform a merge processing operation whenever new file data is generated.

[0076] In a specific implementation, ESS maintains a thread pool, each newly registered application corresponds to a thread, and the application identifier (such as ID) is passed to the thread during the thread startup process; when the application execution is completed, the thread exits by itself.

[0077] In some embodiments, the above thread is driven by a timer. The data structure related to the corresponding application is scanned regularly, and a merge processing operation is performed whenever new file data is generated. Using a timer as a driver can prevent omissions compared to using an event as a driver.

[0078] Figure 3 FIG. 1 is a flow chart of a data processing method provided by another embodiment of the present application. Figure 3 As shown, S202, after merging the to-be-processed data corresponding to the same Reducer to obtain the merged data, the method of the embodiment of the present application may further include:

[0079] S301. Monitor whether an abnormality occurs during the merging process.

[0080] If it is monitored that no abnormality occurs during the merging process, S203 and S204 are executed; if it is monitored that an abnormality occurs during the merging process, S302 and S303 are executed.

[0081] S302: Receive a second read operation sent by a second executor.

[0082] The second read operation is used to read the pre-merged data corresponding to the Reducer.

[0083] S303. According to the second read operation, the pre-merged data corresponding to the Reducer is sent.

[0084] In this embodiment, considering that an exception may occur during the merging process, the exception during the merging process is monitored, and when an exception occurs, the shuffle read process is performed according to the existing process to ensure the normal completion of the corresponding application.

[0085] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0086] Figure 4 This is a schematic diagram of the structure of a data processing device provided in one embodiment of the present application. Figure 4 As shown, the data processing device 40 provided in the embodiment of the present application includes: an acquisition module 41, a processing module 42, a receiving module 43 and a sending module 44. Among them, the acquisition module 41, the receiving module 43 and the sending module 44 are connected to the processing module 42 respectively.

[0087] The acquisition module 41 is used to acquire the data to be processed. The data to be processed is file data written by the first executor, wherein the number of files corresponding to the file data is N times the number of reducers, N is an integer greater than 1, and the reducer is located in the second executor.

[0088] The processing module 42 is used to merge the to-be-processed data corresponding to the same Reducer to obtain merged data, wherein the number of the merged data is less than the number of files corresponding to the file data.

[0089] The receiving module 43 is used to receive a first read operation sent by the second executor. The first read operation is used to read the merged data corresponding to the Reducer.

[0090] The sending module 44 is used to send the merged data corresponding to the Reducer according to the first reading operation.

[0091] Optionally, the number of merged data is equal to the number of Reducers. Those skilled in the art will understand that when the merge process is completed normally, the number of merged data is equal to the number of Reducers; if an exception occurs in the merge process, part of the data to be processed is merged, and the remaining part of the data to be processed remains unchanged. At this time, the number of merged data is greater than the number of Reducers.

[0092] After the merging process is completed normally, in some embodiments, the processing module 42 can also be used to update the data to be processed. That is, during the merging process, the data to be processed needs to be kept unchanged and no update operation is performed on it. Only after the merging process is completed, the data to be processed that has completed the merging process is updated to prevent abnormal phenomena from causing data inconsistency.

[0093] Each merged data corresponds to a block identifier of a corresponding Reducer. In this way, the Reducer can map the block identifier to the corresponding merged data, that is, the shuffle read interface is changed to adapt to the structure of the ESS provided in the embodiment of the present application.

[0094] Furthermore, the data to be processed may be metadata corresponding to at least one application, wherein the processing of metadata corresponding to different applications is independent of each other and does not interfere with each other.

[0095] Optionally, each application corresponds to a thread, and the life cycle of the thread is the same as the life cycle of the corresponding application. That is, when the application is registered, the thread corresponding to the application is generated, and when the application is executed, the thread corresponding to the application is destroyed. The thread is used to periodically scan the data structure related to the corresponding application, and perform a merge processing operation whenever new file data is generated.

[0096] In a specific implementation, the processing module 42 maintains a thread pool, each newly registered application corresponds to a thread, and the identification (eg, ID) of the application is passed to the thread during the thread startup process; when the application execution is completed, the thread exits by itself.

[0097] In some embodiments, the above thread is driven by a timer. The data structure related to the corresponding application is scanned regularly, and a merge processing operation is performed whenever new file data is generated. Using a timer as a driver can prevent omissions compared to using an event as a driver.

[0098] Figure 5 This is a schematic diagram of the structure of a data processing device provided by another embodiment of the present application. Figure 5 As shown, in Figure 4 Based on the structure shown, the data processing device 50 may further include:

[0099] The monitoring module 51 is used to monitor whether an abnormality occurs during the merging process. The monitoring module 51 is connected to the processing module 42 .

[0100] Correspondingly, the receiving module 43 can also be used to receive a second read operation sent by the second executor when the monitoring module 50 detects an abnormality in the merging process, wherein the second read operation is used to read the pre-merged data corresponding to the Reducer.

[0101] The sending module 44 may also be configured to send the pre-merged data corresponding to the Reducer according to the second read operation.

[0102] Since exceptions may occur during the merging process, the monitoring module monitors the exceptions during the merging process, and when an exception occurs, the shuffle read process is carried out according to the existing process to ensure the normal completion of the corresponding application.

[0103] The data processing device provided in the embodiments of the present application can be used to execute the data processing method described in any of the above embodiments of the present application. Its implementation principles and technical effects are similar and will not be repeated here.

[0104] Figure 6 This is a structural diagram of a data processing device provided by another embodiment of the present application. Figure 6 As shown, the data processing device 60 provided in the embodiment of the present application may include: a memory 61 and a processor 62. Among them,

[0105] The memory 61 is used to store program instructions.

[0106] The processor 62 is used to call and execute the program instructions stored in the memory 61. When the processor 62 executes the program instructions, the data processing device 60 implements the data processing method described in any of the above method embodiments. The implementation principle and technical effect are similar and will not be repeated here.

[0107] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the data processing method described in any of the above embodiments of the present application. The implementation principles and technical effects are similar and will not be repeated here.

[0108] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining data to be processed, where the data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, where N is an integer greater than 1, and the Reducer is located in the second executor; The to-be-processed data corresponding to the same Reducer are merged and whether an exception occurs during the merging process is monitored to obtain merged data, where the number of the merged data is less than the number of files corresponding to the file data; If no abnormality occurs during the merging process, a first read operation sent by the second executor is received, where the first read operation is used to read the merged data corresponding to the Reducer; and according to the first read operation, the merged data corresponding to the Reducer is sent; If an exception is detected during the merging process, a second read operation sent by the second executor is received, and the second read operation is used to read the pre-merged data corresponding to the Reducer; according to the second read operation, the pre-merged data corresponding to the Reducer is sent.

2. The method according to claim 1, characterized in that The number of the merged data is equal to the number of the Reducers.

3. The method according to claim 2, characterized in that Each merged data corresponds to the block identifier of the corresponding Reducer.

4. The method according to claim 2, characterized in that After merging the to-be-processed data corresponding to the same Reducer and monitoring whether an exception occurs during the merging process, and obtaining the merged data, the method further includes: The data to be processed is updated.

5. The method according to claim 1, characterized in that The data to be processed is metadata corresponding to at least one application.

6. The method according to claim 5, characterized in that Each of the applications corresponds to a thread. The life cycle of the thread is the same as that of the corresponding application. The thread is used to periodically scan the data structure related to the corresponding application and perform a merging operation whenever new file data is generated.

7. The method according to claim 6, characterized in that The thread is driven by a timer.

8. A data processing device, characterized in that: include: An acquisition module, used for acquiring data to be processed, wherein the data to be processed is file data written by the first executor, the number of files corresponding to the file data is N times the number of Reducers, N is an integer greater than 1, and the Reducer is located in the second executor; A processing module, used for merging the to-be-processed data corresponding to the same Reducer to obtain merged data, wherein the number of the merged data is less than the number of files corresponding to the file data; Monitoring module, used to monitor whether abnormalities occur during the merging process; A receiving module, configured to receive a first read operation sent by the second executor when no abnormality occurs during the merging process, the first read operation being used to read the merged data corresponding to the Reducer; or to receive a second read operation sent by the second executor when an abnormality occurs during the merging process, the second read operation being used to read the pre-merged data corresponding to the Reducer; A sending module is used to send the merged data corresponding to the Reducer according to the first reading operation when no abnormality occurs during the merging process; or to send the pre-merged data corresponding to the Reducer according to the second reading operation when an abnormality occurs during the merging process.

9. The device according to claim 8, characterized in that The number of the merged data is equal to the number of the Reducers.

10. The device according to claim 9, characterized in that The processing module is further used for: After the data to be processed corresponding to the same Reducer are merged to obtain the merged data, the data to be processed are updated.

11. A data processing device, characterized in that: include: Memory and processor; Wherein, the memory is used to store program instructions; The processor is used to call and execute the program instructions. When the processor executes the program instructions, the data processing device implements the method according to any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

13. A computer program product comprising: A computer program, characterized in that when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 7.

Images