Data transmission method, device, computer equipment and storage medium

Abstract

The embodiments of the present disclosure relate to a data transmission method, apparatus, computer equipment and storage medium. By establishing a message queue, when a Map task is completed, the output data of the Map task is added to the message queue, and the output data is transmitted to the corresponding Reduce task according to the order of the output data in the message queue, thereby realizing an asynchronous data transmission method based on the MapReduce framework, improving the data processing efficiency of MapReduce, and solving the problem that network congestion is prone to occur in related technologies.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of big data technology, and in particular, to a data transmission method, apparatus, computer equipment, and storage medium. Background Art

[0002] MapReduce is a distributed computing framework that performs large-scale data processing and computing mainly through "mapping" tasks (or also called Map tasks) and "reduction" tasks (or also called Reduce tasks).

[0003] In the related art, the data transmission process between Map tasks and Reduce tasks is called the "shuffle" process (or the Shuffle process). In this process, only after all Map tasks are completed, the output data of all Map tasks will be written to the local disk at one time. Only after all data is written to the local disk can the Reduce task read data from the disk and execute the corresponding task.

[0004] However, since the Reduce task in the related technology must wait for all Map tasks to be completed before it can be executed, it will lead to low data processing efficiency, and writing a large amount of data to the local disk at one time will also cause network congestion. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a data transmission method, an apparatus, a computer device and a storage medium.

[0006] A first aspect of the present disclosure provides a data transmission method, the method comprising:

[0007] Establish a message queue; in response to monitoring that a Map task is completed, add the output data of the Map task to the message queue; according to the order of the output data of the Map task in the message queue, transmit the output data to the corresponding Reduce task.

[0008] In one implementation, establishing a message queue includes:

[0009] Based on the number of Reduce tasks, a corresponding number of partitions are established; a message queue is established in each partition, and a one-to-one correspondence between the message queue and the Reduce task is established.

[0010] In one implementation, the output data of the Map task includes a field corresponding to the Map task and an execution result of the Map task; and adding the output data of the Map task to the message queue includes:

[0011] According to the field corresponding to the Map task, determine the target message queue corresponding to the Map task, where the target message queue refers to the message queue corresponding to the Reduce task used to process the Map task; add the output data of the Map task to the target message queue.

[0012] In one implementation, determining the target message queue corresponding to the Map task according to the field corresponding to the Map task includes:

[0013] The remainder of the quotient of the hash value of the field and the number of Reduce tasks is calculated; based on the corresponding relationship between the remainder and the message queue, the target message queue corresponding to the Map task is determined.

[0014] In one embodiment, in response to monitoring that a Map task is completed, before adding the output data of the Map task to the message queue, the method further includes:

[0015] A Map task list is established based on all Map tasks, and the Map task list includes the completion status of each Map task and the number of unfinished tasks.

[0016] In one implementation, after adding the output data of the Map task to the message queue, the method further includes:

[0017] Update the number of unfinished tasks in the Map task list.

[0018] In one implementation, after transmitting the output data to the corresponding Reduce task, the method further includes:

[0019] In response to monitoring that the number of unfinished tasks in the Map task list is 0, all message queues are closed.

[0020] In one implementation, before transmitting the output data to the corresponding Reduce task, the method further includes:

[0021] Detecting whether the Reduce task is in an idle state or has finished running; transmitting the output data to the corresponding Reduce task includes:

[0022] If the Reduce task is in an idle state or has finished running, the output data is transmitted to the corresponding Reduce task. A second aspect of the embodiment of the present disclosure provides a data transmission device, which includes:

[0023] The first establishing module is used to establish a message queue.

[0024] The data adding module is used to add the output data of the Map task to the message queue when it is detected that a Map task is completed.

[0025] The data transmission module is used to transmit the output data of the Map task to the corresponding Reduce task according to the order of the output data in the message queue.

[0026] In one embodiment, the first establishing module is used to:

[0027] Based on the number of Reduce tasks, a corresponding number of partitions are established; a message queue is established in each partition, and a one-to-one correspondence between the message queue and the Reduce task is established.

[0028] In one implementation, the output data of the Map task includes a field corresponding to the Map task and an execution result of the Map task;

[0029] The data adding module comprises:

[0030] A determining unit, configured to determine a target message queue corresponding to the Map task according to the field corresponding to the Map task, wherein the target message queue refers to a message queue corresponding to a Reduce task for processing the Map task;

[0031] The adding unit is used to add the output data of the Map task to the target message queue.

[0032] In one implementation, the determining unit is configured to:

[0033] The remainder of the quotient of the hash value of the field and the number of Reduce tasks is calculated; and based on the corresponding relationship between the remainder and the message queue, the target message queue corresponding to the Map task is determined.

[0034] In one embodiment, the device further comprises:

[0035] The second establishment module is used to establish a Map task list based on all Map tasks before detecting that a Map task is completed and adding the output data of the Map task to the message queue. The Map task list includes the completion status of each Map task and the number of unfinished tasks.

[0036] In one embodiment, the device further comprises:

[0037] An updating module is used to update the number of unfinished tasks in the Map task list after adding the output data of the Map task to the message queue.

[0038] In one embodiment, the device further comprises:

[0039] The monitoring module is used to close all message queues when it is detected that the number of unfinished tasks in the Map task list is 0.

[0040] In one embodiment, the device further comprises:

[0041] A detection module, used for detecting whether the Reduce task is in an idle state or has finished running before transmitting the output data to the corresponding Reduce task;

[0042] The data transmission module is used to transmit the output data to the corresponding Reduce task when the Reduce task is in an idle state or has finished running. A third aspect of the embodiment of the present disclosure provides a computer device, the computer device comprising: a memory and a processor; wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of the first aspect is implemented.

[0043] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0044] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0045] By establishing a message queue, whenever a Map task is completed, the output data of the Map task is added to the message queue, and the output data is transmitted to the corresponding Reduce task according to the order of the output data in the message queue, thereby realizing an asynchronous data transmission method based on the MapReduce framework. This method does not need to wait for all Map tasks to be completed, and can transmit the output data of a single completed Map task to the corresponding Reduce task in the form of a message queue for processing, thereby saving the waiting time of the Reduce task and improving the data processing efficiency in MapReduce. In addition, since the output data of all Map tasks in the embodiment of the present disclosure are waiting to be sent in the transmission pipeline of the message queue, there is no need to write the output data of all Map tasks to the disk at one time as in the related technology, thereby avoiding the network congestion caused by a one-time write. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a data transmission method based on the MapReduce framework provided for related technologies;

[0049] Figure 2 is a flow chart of a data transmission method provided by an embodiment of the present disclosure;

[0050] Figure 3 It is a schematic diagram of a data transmission method based on the MapReduce framework provided in an embodiment of the present disclosure;

[0051] Figure 4 is a flow chart of a method for establishing a message queue provided by an embodiment of the present disclosure;

[0052] Figure 5 It is a schematic diagram of a data transmission method based on the MapReduce framework provided in an embodiment of the present disclosure;

[0053] Figure 6 is a flow chart of a method for establishing a message queue provided by an embodiment of the present disclosure;

[0054] Figure 7 is a structural schematic diagram of a data transmission device provided by an embodiment of the present disclosure;

[0055] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0058] Figure 1 A schematic diagram of a data transmission method based on the MapReduce framework provided for related technologies, such as Figure 1 As shown, N Map tasks and M Reduce tasks can be pre-set in the MapReduce framework. The number of Map tasks N and the number of Reduce tasks M can be the same or different. The values ​​of N and M are positive integers.

[0059] In the related art, Map tasks and Reduce tasks are generally used for data statistics. For example, when a data file is input into the MapReduce framework, it will first be divided into multiple data slices. A Map task is used to perform statistical processing on the data in a data slice, such as counting a certain word in the data slice, or the number of characters in the data slice. During the execution of the Map task, all Reduce tasks do not perform data processing. When all Map tasks are completed, the output data of all Map tasks are written to the disk at one time to avoid frequent disk write operations and increase port overhead. After monitoring that all Map tasks have been completed, the Reduce task in MapReduce reads the output data of each corresponding Map task from the disk, and then performs total statistics and aggregation on the output data of one or more Map tasks obtained. For example, the number of occurrences of a specific word counted by each Map task is aggregated to obtain the total number of occurrences of the word in the data file, or the number of words counted by each Map task is aggregated to obtain the total number of words in the data file, etc.

[0060] exist Figure 1 In the related technology provided, in order to reduce the number of disk write operations and reduce port overhead, a method of one-time data writing is adopted. However, this method needs to wait for all Map tasks to be completed before the output data of all Map tasks can be written to the disk at one time. Before this, the data of any Map task cannot be written to the disk. All Reduce tasks must wait for all Map tasks to be completed before they can obtain the corresponding data from the disk. This greatly increases the data processing time and reduces the data processing efficiency. At the same time, since the amount of data written at one time is usually large, it often causes network congestion when performing write operations.

[0061] In response to the problems existing in the related technologies, the embodiments of the present disclosure provide a data transmission solution. The solution establishes a message queue to asynchronously transmit the output data of each Map task to the Reduce task in the form of a message queue, so that the Reduce task can perform the corresponding processing without waiting for all Map tasks to be completed, thereby improving the data processing efficiency and avoiding network congestion caused by one-time data access.

[0062] In order to more clearly understand the technical solution of the embodiment of the present disclosure, the technical solution of the embodiment of the present disclosure is explained below in conjunction with exemplary embodiments.

[0063] Figure 2 1 is a flow chart of a data transmission method provided by an embodiment of the present disclosure, and the method is applicable to a MapReduce framework. Figure 2 As shown, the method includes:

[0064] Step 201: Establish a message queue.

[0065] Step 202: In response to monitoring that a Map task is completed, the output data of the Map task is added to the message queue.

[0066] Step 203: According to the order of the output data of the Map task in the message queue, the output data is transmitted to the corresponding Reduce task.

[0067] The "message queue" referred to in this embodiment is a container for storing messages during the transmission of messages. The data in the message queue is sent in sequence according to the order of the data in the message queue.

[0068] For example, Figure 3 is a schematic diagram of a data transmission method based on the MapReduce framework provided by an embodiment of the present disclosure. Figure 3 It includes N Map tasks and M Reduce tasks. There is a corresponding relationship between Map tasks and Reduce tasks, that is, the output data of each Map task is processed by a pre-specified Reduce task, and a Reduce task can process the output data of one or more Map tasks. N and M are positive integers.

[0069] like Figure 3 As shown, in the MapReduce framework, the data file input to the MapReduce framework will be processed by data sharding to obtain multiple data shards. The specific number of data shards obtained is determined by the data volume of the data file itself. For example, if the data file is 1280M in total and the upper limit of the data volume of each shard is set to 128M, then when performing data sharding, each 128M of data can be divided into one data shard, and a total of 10 data shards are obtained. Of course, this is only an example, not the only limitation on the data sharding method in the embodiment of the present disclosure.

[0070] In actual situations, the number of Map tasks is generally set to be greater than or equal to the number of data shards. Figure 3The example in the figure configures the number of data shards to be equal to the number of Map tasks N. In this embodiment, a Map task processes one data shard at a time. The output data of the Map task includes the field key corresponding to the Map task and the execution result value of the Map task (for example, the number of words counted by the Map task from the data shard, or the number of characters in the data shard, etc.), where the field key corresponding to the Map task is a preset value, which is used to indicate which Reduce task is responsible for processing the output data of the Map task.

[0071] After the Map task is executed, the output data of the task is added to the message queue, and the output data will be sent in the message queue according to its order in the message queue. When executing the sending operation, the output data can be sent to the Reduce task corresponding to the key value according to the specific value of the key in the output data. Therefore, the Reduce task can obtain the data and perform the corresponding processing operation without waiting for all Map tasks to be completed.

[0072] Alternatively, in one implementation, before sending the output data to the corresponding Reduce task, the running state of the Reduce task can be monitored to determine whether the Reduce task is in an idle state or has finished running. If so, the output data can be directly sent to the Reduce task. Otherwise, the output data can be sent to the Reduce task when the Reduce task is in an idle state or has finished running. This ensures that the output data can be accurately received and prevents the Reduce task from missing the output data.

[0073] In this embodiment, a message queue is established so that whenever a Map task is completed, the output data of the Map task is added to the message queue, and the output data is transmitted to the corresponding Reduce task according to the order of the output data in the message queue, thereby realizing an asynchronous data transmission method based on the MapReduce framework. This method does not need to wait for all Map tasks to be completed, and can transmit the output data of a single completed Map task to the corresponding Reduce task in the form of a message queue for processing, thereby saving the waiting time of the Reduce task and improving the data processing efficiency in MapReduce. In addition, since the output data of all Map tasks in this embodiment are waiting to be sent in the transmission pipeline of the message queue, there is no need to write the output data of all Map tasks to the disk at one time as in the related technology, thereby avoiding the network incurred by a one-time write.

[0074] Figure 4FIG. 1 is a flow chart of a method for establishing a message queue provided by an embodiment of the present disclosure. Figure 4 As shown, the method comprises the following steps:

[0075] Step 401: Create a corresponding number of partitions based on the number of Reduce tasks.

[0076] Step 402: Establish a message queue in each partition, and establish a one-to-one correspondence between the message queue and the Reduce task.

[0077] The partitions referred to in this embodiment can be exemplarily understood as Kafka partitions obtained based on Kafka technology, but are not limited to Kafka partitions. The Kafka technology used in this embodiment is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers.

[0078] The Kafka method adopted in this embodiment is only one of many methods that can be used to establish a message queue. In other implementation methods, other methods in related technologies can be used to establish the message queue in the embodiment of the present disclosure. Here, only Kafka is used as an example for explanation, and the method of establishing a message queue in other ways can refer to the Kafka method, which will not be elaborated here.

[0079] For example, Figure 5 Schematic diagram of a data transmission method based on the MapReduce framework provided by an embodiment of the present disclosure. Figure 5It includes N Map tasks and M Reduce tasks. Among them, a corresponding message queue is established for each Reduce task, and a one-to-one correspondence between the message queue and the Reduce task is obtained. In this embodiment, each message queue has a unique identifier. For example, in one implementation, the identifier of the message queue can be specifically [I, R_position], where I is the index of the message queue, and R_position is used to determine the correspondence between the Map task and the message queue. For example, in one implementation, R_position can be specifically the hash value of the key value. At this time, after obtaining the output data of the Map task, the hash value corresponding to the key value in the output data can be calculated, and the message queue containing the hash value in the identifier is determined as the target message queue corresponding to the Map task. For another example, in another implementation, R_position can be specifically the key value itself. At this time, the key value output by the Map task can be directly compared with the R_position of each message queue, and the message queue with consistent comparison is determined as the target message queue corresponding to the Map task. For another example, in another implementation, R_position can also be specifically defined as the remainder of the quotient of the hash value of the key value and the number of Reduce tasks. In this case, after obtaining the output data of the Map task, the remainder of the quotient of the hash value of the key value in the output data and the number of Reduce tasks can be calculated, and the target message queue corresponding to the Map task can be determined through the correspondence between the remainder and the message queue (the remainder corresponds to the message queue including the remainder in the identifier). That is to say, in this embodiment, the target message queue corresponding to the Map task can be determined according to the field key corresponding to the Map task, and the output data of the Map task can be added to the target message queue for sending, wherein the target message queue refers to the message queue corresponding to the Reduce task used to process the Map task.

[0080] Of course, the above description about R_position in this embodiment is only an example and is not the only limitation of the embodiment of the present disclosure.

[0081] In this embodiment, by establishing a corresponding message queue for each Reduce task, the Reduce tasks in the MapReduce framework can process data in parallel without waiting between the Reduce tasks, thereby further improving the data processing efficiency of the MapReduce framework.

[0082] Figure 6 FIG. 1 is a flow chart of a method for establishing a message queue provided by an embodiment of the present disclosure. Figure 6 As shown, the method provided in this embodiment includes the following steps:

[0083] Step 601: Establish a message queue, and establish a Map task list based on all Map tasks. The Map task list includes the completion status of each Map task and the number of unfinished tasks.

[0084] Step 602: In response to monitoring that a Map task is completed, the output data of the Map task is added to the message queue.

[0085] Step 603: Update the number of unfinished tasks in the Map task list.

[0086] Step 604: According to the order of the output data of the Map task in the message queue, the output data is transmitted to the corresponding Reduce task.

[0087] The order between step 603 and step 604 can be arbitrary.

[0088] Step 605: In response to monitoring that the number of unfinished tasks in the Map task list is 0, all message queues are closed.

[0089] This embodiment establishes a Map task list and maintains the completion status of all Map tasks through the Map task list, so as to accurately obtain the completion status of all Map tasks, so as to close all message queues in time when all Map tasks are completed to avoid affecting subsequent tasks.

[0090] Figure 7 is a structural diagram of a data transmission device provided by an embodiment of the present disclosure, such as Figure 7 As shown, the data transmission device 70 includes:

[0091] The first establishing module 71 is used to establish a message queue.

[0092] The data adding module 72 is used to add the output data of the Map task to the message queue when it is detected that a Map task is completed.

[0093] The data transmission module 73 is used to transmit the output data to the corresponding Reduce task according to the order of the output data in the message queue.

[0094] In a possible implementation, the first establishing module 71 may be used to:

[0095] Based on the number of Reduce tasks, a corresponding number of partitions are established; a message queue is established in each partition, and a one-to-one correspondence between the message queue and the Reduce task is established.

[0096] In a possible implementation manner, the output data of the Map task includes fields corresponding to the Map task and the execution result of the Map task; the data adding module 72 includes:

[0097] A determination unit, configured to determine a target message queue corresponding to the Map task according to the field corresponding to the Map task, wherein the target message queue refers to a message queue corresponding to a Reduce task for processing the Map task.

[0098] The adding unit is used to add the output data of the Map task to the target message queue.

[0099] In a possible implementation manner, the determining unit is configured to:

[0100] Calculate the remainder of the quotient of the hash value of the field and the number of Reduce tasks; and

[0101] Based on the corresponding relationship between the remainder and the message queue, the target message queue corresponding to the Map task is determined.

[0102] In a possible implementation, the device further includes:

[0103] The second establishment module is used to establish a Map task list based on all Map tasks before detecting that a Map task is completed and adding the output data of the Map task to the message queue. The Map task list includes the completion status of each Map task and the number of unfinished tasks.

[0104] In a possible implementation, the device further includes:

[0105] An updating module is used to update the number of unfinished tasks in the Map task list after adding the output data of the Map task to the message queue.

[0106] In a possible implementation, the device further includes:

[0107] The monitoring module is used to close all message queues when it is detected that the number of unfinished tasks in the Map task list is 0.

[0108] In a possible implementation, the device further includes:

[0109] A detection module, used for detecting whether the Reduce task is in an idle state or has finished running before transmitting the output data to the corresponding Reduce task;

[0110] The data transmission module is used to transmit the output data to the corresponding Reduce task when the Reduce task is in an idle state or has finished running.

[0111] The device provided in this embodiment can perform the above Figure 2-Figure 6 The method of any embodiment has similar execution mode and beneficial effects, which will not be described in detail here.

[0112] Figure 8 is a schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure, such as Figure 8 As shown, the computer device 80 includes: a memory 81 and a processor 82;

[0113] The memory 81 stores a computer program, which, when executed by the processor 82, implements the above Figure 2-Figure 6 The method of any embodiment has similar execution mode and beneficial effects, which will not be described in detail here.

[0114] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above Figure 2-Figure 6 The method of any embodiment has similar execution mode and beneficial effects, which will not be described in detail here.

[0115] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0116] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data transmission method, characterized in that: include: Create a message queue; In response to monitoring that a Map task is completed, adding the output data of the Map task to the message queue; According to the order of the output data in the message queue, the output data is transmitted to the corresponding Reduce task; In response to monitoring that a Map task is completed, before adding the output data of the Map task to the message queue, the method further includes: Establishing a Map task list based on all Map tasks, wherein the Map task list includes the completion status of each Map task and the number of unfinished tasks; After adding the output data of the Map task to the message queue, the method further includes: Update the number of unfinished tasks in the Map task list; After transmitting the output data to the corresponding Reduce task, the method further includes: In response to monitoring that the number of unfinished tasks in the Map task list is 0, all message queues are closed.

2. The method according to claim 1, characterized in that The establishing of the message queue comprises: Based on the number of Reduce tasks, create a corresponding number of partitions; A message queue is established in each partition, and a one-to-one correspondence between the message queue and the Reduce task is established.

3. The method according to claim 2, characterized in that The output data of the Map task includes the fields corresponding to the Map task and the execution result of the Map task; The adding the output data of the Map task to the message queue includes: Determine, according to the field corresponding to the Map task, a target message queue corresponding to the Map task, wherein the target message queue refers to a message queue corresponding to a Reduce task for processing the Map task; The output data of the Map task is added to the target message queue.

4. The method according to claim 3, characterized in that The determining, according to the field corresponding to the Map task, a target message queue corresponding to the Map task, includes: Calculate the remainder of the quotient of the hash value of the field and the number of Reduce tasks; Based on the corresponding relationship between the remainder and the message queue, the target message queue corresponding to the Map task is determined.

5. The method according to any one of claims 1 to 4, characterized in that Before transmitting the output data to the corresponding Reduce task, the method further includes: Detect whether the Reduce task is in an idle state or has finished running; The transmitting the output data to the corresponding Reduce task includes: If the Reduce task is in an idle state or has finished running, the output data is transmitted to the corresponding Reduce task.

6. A data transmission device, characterized in that: include: The first establishment module is used to establish a message queue; A data adding module, used for adding the output data of the Map task to the message queue when monitoring that a Map task is completed; A data transmission module, used for transmitting the output data to a corresponding Reduce task according to the order of the output data in the message queue; A second establishing module is used to establish a Map task list based on all Map tasks before detecting that a Map task is completed and adding the output data of the Map task to the message queue, wherein the Map task list includes the completion status of each Map task and the number of unfinished tasks; An updating module, configured to update the number of unfinished tasks in the Map task list after adding the output data of the Map task to the message queue; The monitoring module is used to close all message queues when it is detected that the number of unfinished tasks in the Map task list is 0.

7. The device according to claim 6, characterized in that The first establishing module is used to: Based on the number of Reduce tasks, create a corresponding number of partitions; A message queue is established in each partition, and a one-to-one correspondence between the message queue and the Reduce task is established.

8. The device according to claim 7, characterized in that The output data of the Map task includes the fields corresponding to the Map task and the execution result of the Map task; The data adding module comprises: A determining unit, configured to determine a target message queue corresponding to the Map task according to the field corresponding to the Map task, wherein the target message queue refers to a message queue corresponding to a Reduce task for processing the Map task; The adding unit is used to add the output data of the Map task to the target message queue.

9. The device according to claim 8, characterized in that The determining unit is used for: Calculate the remainder of the quotient of the hash value of the field and the number of Reduce tasks; and Based on the corresponding relationship between the remainder and the message queue, the target message queue corresponding to the Map task is determined.

10. The device according to any one of claims 6 to 9, characterized in that: The device also includes: A detection module, used for detecting whether the Reduce task is in an idle state or has finished running before transmitting the output data to the corresponding Reduce task; The data transmission module is used to transmit the output data to the corresponding Reduce task when the Reduce task is in an idle state or has finished running.

11. A computer device, characterized in that: include: Memory and processor; The memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images