A data processing method and apparatus

Abstract

This application provides a data processing method and apparatus for improving the execution efficiency of shuffle operations and the operating efficiency of distributed computing frameworks. The method includes: performing a first operation on N key-value data from a first application, and determining a first duration for the first operation, wherein the first operation is an aggregation operation and / or a sorting operation; if the first duration exceeds a first threshold, not performing a second operation on the remaining key-value data from the first application other than the N key-value data; wherein the first operation is the aggregation operation, and the second operation is the aggregation operation; or, the first operation is the sorting operation, and the second operation is the sorting operation; or the first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] With the development of computer technology, distributed computing frameworks such as MapReduce and Spark may need to aggregate data with common characteristics onto a single node for computation based on business requirements. Therefore, distributed computing frameworks may need to perform data shuffling during computation. A distributed computing framework consists of multiple map nodes and multiple reduce nodes. Each map node can communicate with the multiple reduce nodes, and each reduce node can communicate with the multiple map nodes.

[0003] During the shuffle operation, each of the multiple map nodes can shuffle and repartition the data to be processed based on the data processing range of the reduce nodes, and store the data belonging to different reduce nodes in separate partitions. Each of the multiple reduce nodes can retrieve the data of the corresponding partition from the storage area of ​​each map node, and then reassemble the data retrieved from the multiple map nodes. The shuffle operation is integrated into the execution of computation tasks within the distributed computing framework; therefore, the execution efficiency of the shuffle operation affects the overall operating efficiency of the distributed computing framework.

[0004] The data processing in a shuffle operation is divided into two stages: the map stage and the reduce stage. In the map stage, map nodes perform data processing, and in the reduce stage, reduce nodes perform data processing. During the shuffle operation, the data processed by the map and reduce nodes is, for example, key-value (KV) data. For data from certain applications, there may be a need for aggregation and / or sorting. Aggregation is used to merge values ​​corresponding to the same key from multiple KV entries, and sorting is used to sort the KV entries according to the keys they contain. In this context, the application's aggregation and / or sorting requirements indicate that reduce nodes need to perform aggregation and / or sorting operations, while map nodes may or may not. That is, aggregation and / or sorting requirements only require reduce nodes to perform aggregation and / or sorting operations, but not map nodes. Whether map nodes perform aggregation and / or sorting operations can be determined by the distributed computing framework. For example, some distributed computing frameworks support aggregation operations on map nodes, so each map node will perform aggregation operations during the map phase, while other distributed computing frameworks do not support aggregation operations on map nodes, so map nodes will not perform aggregation operations.

[0005] However, for some applications, performing aggregation and / or sorting operations on the data during the map phase can actually reduce the efficiency of the shuffle operation. If the distributed computing framework performs aggregation and / or sorting operations on the data of these applications during the map phase, it may reduce the efficiency of the shuffle operation, thereby reducing the overall efficiency of the distributed computing framework. Summary of the Invention

[0006] This application provides a data processing method and apparatus for improving the execution efficiency of shuffle operations and the operating efficiency of distributed computing frameworks.

[0007] Firstly, a data processing method is provided, which can be implemented by a first mapping node among multiple mapping nodes included in a distributed computing framework. The method includes: performing a first operation on N key-value data from a first application, and determining a first duration for the first operation. Each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword. The first operation is an aggregation operation and / or a sorting operation. The aggregation operation is used to merge values ​​corresponding to the same keyword in the N key-value data, and the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1. If the first duration is greater than a first threshold, a second operation is not performed on the remaining key-value data from the first application other than the N key-value data. Wherein, the first operation is the aggregation operation, and the second operation is the aggregation operation; or, the first operation is the sorting operation, and the second operation is the sorting operation; or, the first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

[0008] In this embodiment, the first mapping node can first perform aggregation and / or sorting operations on N key-value data from the first application and determine the duration of the operation. If the duration exceeds a first threshold, it indicates that the operation is time-consuming and may reduce the efficiency of the shuffle operation. Therefore, the first mapping node may not perform a second operation on the remaining key-value data from the first application other than the one that has already undergone the first operation. This reduces the duration of the shuffle operation, improves the execution efficiency of the shuffle operation, and thus improves the operating efficiency of the distributed computing framework.

[0009] In one possible implementation, the first operation is the aggregation operation, and determining the first duration of the first operation includes: determining the difference between a second duration and a third duration as the first duration, wherein the third duration is the predicted duration for writing the N key-value data into the first storage area; or, determining the second duration as the first duration; wherein the second duration is the actual duration for writing the N key-value data into the first storage area after the aggregation operation.

[0010] In the above technical solution, the first duration, determined by combining the actual time taken to write the N key-value data to the first storage area after performing an aggregation operation and the predicted time taken to write the N key-value data to the first storage area without performing an aggregation operation, can improve the accuracy of determining the first duration, thereby improving the accuracy of determining whether an aggregation operation needs to be performed on the key-value data to be processed. Furthermore, directly using the actual time taken to write the N key-value data to the first storage area after performing an aggregation operation as the first duration can shorten the time required to determine the first duration, improve the efficiency of the shuffle operation, and thus improve the operating efficiency of the distributed computing framework.

[0011] In one possible implementation, the first operation is the aggregation operation, and the method further includes: if the first duration is less than or equal to the first threshold and greater than the second threshold, determining the number M of key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of M to N is greater than the third threshold, not performing the aggregation operation on the remaining key-value data from the first application other than the N key-value data.

[0012] In the above technical solution, if the duration of the aggregation operation is less than or equal to the first threshold and greater than the second threshold, it indicates that the aggregation operation has little impact on the duration of the shuffle operation. Therefore, the amount of key-value data reduced by the aggregation operation can also be determined. If the amount of key-value data reduced by the aggregation operation is small, the number of times the map node accesses the storage space can be reduced, thereby reducing the input / output (I / O) overhead. Therefore, the aggregation operation can be performed on the key-value data to be processed. However, if the amount of key-value data reduced by the aggregation operation is also small, the I / O overhead is still large, and the time for the map node to write data to the storage space needs to be increased. Therefore, the aggregation operation can be avoided for the key-value data to be processed, reducing the probability of increasing the duration of the shuffle operation.

[0013] In one possible implementation, the first operation is the aggregation operation, and the method further includes: if the first duration is less than or equal to the first threshold and greater than the second threshold, determining the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, not performing the aggregation operation on the remaining key-value data from the first application other than the N key-value data, where the second storage capacity is the storage capacity occupied by the N key-value data.

[0014] In the above technical solution, it is also possible to determine the storage capacity occupied by the key-value data obtained after the aggregation operation. If the storage capacity occupied by the key-value data after the aggregation operation is reduced significantly, the number of times the map node accesses the storage space can be reduced, thereby reducing I / O overhead. Therefore, the aggregation operation can be performed on the key-value data to be processed. However, if the storage capacity occupied by the key-value data after the aggregation operation is reduced only slightly, the I / O overhead is still large, and the time required for the map node to write data to the storage space needs to be increased. Therefore, the aggregation operation can be avoided for the key-value data to be processed, reducing the probability of increasing the shuffle operation time.

[0015] In one possible implementation, the first operation is the sorting operation, or the first operation is the sorting operation and the aggregation operation, wherein each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0016] In the above technical solution, the key-value data used for sorting carries a first identifier to indicate that it has been sorted, thereby reducing the probability that the reduce node will perform repeated sorting when it reads key-value data from different map nodes.

[0017] Secondly, a data processing method is provided, which can be implemented by a first mapping node among multiple mapping nodes included in a distributed computing framework. The method includes: performing a first operation on N key-value data from a first application and determining a first duration for the first operation, wherein each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword, the first operation being an aggregation operation or a sorting operation, wherein the aggregation operation is used to merge values ​​corresponding to the same keyword in the N key-value data, and the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1; if the first duration is greater than a first threshold, outputting the remaining key-value data from the first application other than the N key-value data.

[0018] In this embodiment, the remaining key-value data output from the first application, excluding the N key-value data that have undergone the first operation, is not subjected to the first operation. Instead, the remaining key-value data is directly output to the corresponding location, such as being written to the corresponding storage area. For example, if the first operation is an aggregation operation, and the first duration of the first operation exceeds a first threshold, the remaining key-value data is not aggregated; it is directly written to the first storage space. If the first operation is a sorting operation, and the first duration of the first operation exceeds the first threshold, the remaining key-value data is not sorted; it is directly written to the second storage space. This second storage space, for example, is a local storage space used to store the repartitioned key-value data.

[0019] In one possible implementation, the first operation is the aggregation operation, and determining the first duration of the first operation includes: determining the difference between a second duration and a third duration as the first duration, wherein the third duration is the predicted duration for writing the N key-value data into the first storage area; or, determining the second duration as the first duration; wherein the second duration is the actual duration for writing the N key-value data into the first storage area after the aggregation operation.

[0020] In one possible implementation, the first operation is the aggregation operation, and the method further includes: if the first duration is less than or equal to the first threshold and greater than the second threshold, determining the number M of key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of M to N is greater than the third threshold, outputting the remaining key-value data from the first application other than the N key-value data.

[0021] In one possible implementation, the first operation is the aggregation operation, and the method further includes: if the first duration is less than or equal to the first threshold and greater than the second threshold, determining the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, outputting the remaining key-value data from the first application other than the N key-value data, where the second storage capacity is the storage capacity occupied by the N key-value data.

[0022] In one possible implementation, the first operation is the sorting operation, wherein each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0023] Thirdly, a data processing method is provided, which can be implemented by a first mapping node among multiple mapping nodes included in a distributed computing framework. The method includes: performing a first operation on N key-value data from a first application and determining a first duration for the first operation, wherein each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword, the first operation being an aggregation operation and a sorting operation, wherein the aggregation operation is used to merge values ​​corresponding to the same keyword in the N key-value data, and the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1; if the first duration is greater than a first threshold, outputting the remaining key-value data from the first application other than the N key-value data, or performing a second operation on the remaining key-value data from the first application other than the N key-value data, wherein the second operation is the aggregation operation or the sorting operation.

[0024] In this embodiment, the remaining key-value data from the first application, excluding the N key-value data, is not aggregated or sorted. Instead, it is directly written to a corresponding location, such as a second storage space, which is, for example, local storage used to store the repartitioned key-value data. A second operation is performed on the remaining key-value data from the first application. If the second operation is an aggregation operation, it only aggregates the remaining key-value data without sorting it; if the second operation is a sorting operation, it only sorts the remaining key-value data without aggregation.

[0025] In one possible implementation, determining the first duration of the first operation includes: determining the difference between the second duration and the third duration as a first sub-duration, or determining the second duration as the first sub-duration; wherein the second duration is the actual duration of writing the N key-value data into the first storage area after the aggregation operation, the third duration is the predicted duration of writing the N key-value data into the first storage area, and the first sub-duration is the duration of the aggregation operation; determining the second sub-duration, which is the duration of the sorting operation; and determining the sum of the first sub-duration and the second sub-duration as the first duration.

[0026] In one possible implementation, each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0027] Fourthly, a data processing apparatus is provided, which has the functions described in the method example of the first aspect above. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. This device can be deployed with a first mapping node among multiple mapping nodes included in a distributed computing framework. The device includes a processing module for performing a first operation on N key-value data from a first application and determining a first duration of the first operation. If the first duration exceeds a first threshold, a second operation is not performed on the remaining key-value data from the first application besides the N key-value data. Each of the N key-value data includes a keyword and a value corresponding to the keyword. The first operation is an aggregation operation and / or a sorting operation. The aggregation operation merges values ​​corresponding to the same keyword, and the sorting operation sorts the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1. Specifically, the first operation is the aggregation operation, and the second operation is the aggregation operation; or the first operation is the sorting operation, and the second operation is the sorting operation; or the first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

[0028] In one possible implementation, the first operation is the aggregation operation, and the processing module is configured to determine the first duration of the first operation by: determining the difference between the second duration and the third duration as the first duration, wherein the third duration is the predicted duration of writing the N key-value data into the first storage area; or, determining the second duration as the first duration; wherein the second duration is the actual duration of writing the N key-value data into the first storage area after the aggregation operation.

[0029] In one possible implementation, the first operation is the aggregation operation, and the processing module is further configured to: determine the number M of key-value data obtained after performing the aggregation operation on the N key-value data when the first duration is less than or equal to the first threshold and greater than the second threshold, where M is a positive integer less than N; if the ratio of M to N is greater than the third threshold, not perform the aggregation operation on the remaining key-value data from the first application other than the N key-value data.

[0030] In one possible implementation, the first operation is the aggregation operation, and the processing module is further configured to: if the first duration is less than or equal to the first threshold and greater than the second threshold, determine the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, not perform the aggregation operation on the remaining key-value data from the first application other than the N key-value data, where the second storage capacity is the storage capacity occupied by the N key-value data.

[0031] In one possible implementation, the first operation is the sorting operation, or the first operation is the sorting operation and the aggregation operation, wherein each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0032] Fifthly, a data processing apparatus is provided, which has the functions described in the method example of the second aspect above. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. The apparatus can be deployed with a first mapping node among multiple mapping nodes included in a distributed computing framework. The apparatus includes: a processing module, configured to perform a first operation on N key-value data from a first application, and determine a first duration of the first operation; if the first duration is greater than a first threshold, output the remaining key-value data from the first application excluding the N key-value data, wherein each of the N key-value data includes a keyword and a value corresponding to the keyword; the first operation is an aggregation operation and / or a sorting operation; the aggregation operation is used to merge values ​​corresponding to the same keyword among the N key-value data; the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1.

[0033] In one possible implementation, the first operation is the aggregation operation, and the processing module is specifically configured to: determine the difference between the second duration and the third duration as the first duration, wherein the third duration is the predicted duration for writing the N key-value data into the first storage area; or, determine the second duration as the first duration; wherein the second duration is the actual duration for writing the N key-value data into the first storage area after the aggregation operation.

[0034] In one possible implementation, the first operation is the aggregation operation, and the processing module is further configured to: determine the number M of key-value data obtained after performing the aggregation operation on the N key-value data when the first duration is less than or equal to the first threshold and greater than the second threshold, where M is a positive integer less than N; and output the remaining key-value data from the first application other than the N key-value data if the ratio of M to N is greater than the third threshold.

[0035] In one possible implementation, the first operation is the aggregation operation, and the processing module is further configured to: if the first duration is less than or equal to the first threshold and greater than the second threshold, determine the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, output the remaining key-value data from the first application other than the N key-value data, where the second storage capacity is the storage capacity occupied by the N key-value data.

[0036] In one possible implementation, the first operation is the sorting operation, wherein each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0037] Sixthly, a data processing apparatus is provided, which has the functions of the method example in the third aspect described above. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. The apparatus can be deployed with a first mapping node among multiple mapping nodes included in a distributed computing framework. The apparatus includes: a processing module, configured to perform a first operation on N key-value data from a first application, and determine a first duration of the first operation; if the first duration is greater than a first threshold, output the remaining key-value data from the first application excluding the N key-value data; or, perform a second operation on the remaining key-value data from the first application excluding the N key-value data, the second operation being the aggregation operation or the sorting operation, wherein each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword; the first operation is an aggregation operation and / or a sorting operation; the aggregation operation is used to merge values ​​corresponding to the same keyword in the N key-value data; the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1.

[0038] In one possible implementation, the processing module is specifically configured to: determine the difference between the second duration and the third duration as a first sub-duration, or determine the second duration as the first sub-duration; wherein the second duration is the actual duration for writing the N key-value data into the first storage area after the aggregation operation, the third duration is the predicted duration for writing the N key-value data into the first storage area, and the first sub-duration is the duration experienced by the aggregation operation; determine the second sub-duration, which is the duration experienced by the sorting operation; and determine the sum of the first sub-duration and the second sub-duration as the first duration.

[0039] In one possible implementation, each of the N key-value data carries a first tag after the sorting operation, the first tag indicating that it has been sorted.

[0040] In a seventh aspect, embodiments of this application provide a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory;

[0041] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform any of the possible methods in the first aspect, or to cause the cluster of computing devices to perform any of the possible methods in the second aspect, or to cause the cluster of computing devices to perform any of the possible methods in the third aspect.

[0042] Eighthly, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform any of the possible methods described in the first aspect, or cause the cluster of computing devices to perform any of the possible methods described in the second aspect, or cause the cluster of computing devices to perform any of the possible methods described in the third aspect.

[0043] Ninthly, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform any of the possible methods in the first aspect, or cause the cluster of computing devices to perform any of the possible methods in the second aspect, or cause the cluster of computing devices to perform any of the possible methods in the third aspect.

[0044] In a tenth aspect, a mapping server is provided, the mapping server comprising: one or more processors; one or more memories; wherein the one or more memories store one or more computer instructions, which, when executed by the one or more processors, cause the mapping server to perform the method as described in any one of the first aspects above, or cause the mapping server to perform the method as described in any one of the second aspects above, or cause the mapping server to perform the method as described in any one of the third aspects above.

[0045] In one aspect, this application provides a chip, the chip including a processor coupled to a memory, for reading and executing a software program stored in the memory to implement the method described in any one of the first aspects, or the method described in any one of the second aspects, or the method described in any one of the third aspects.

[0046] For the beneficial effects described in the second to eleventh aspects above, please refer to the beneficial effects in the first aspect; they will not be repeated here. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the architecture of a distributed computing framework;

[0048] Figure 2 This is a schematic diagram of the architecture of another distributed computing framework;

[0049] Figure 3 A flowchart for a type of data processing;

[0050] Figure 4 This is a schematic diagram of the architecture of a distributed processing system provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the actuator architecture provided in an embodiment of this application;

[0052] Figure 6 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0053] Figure 7 A flowchart illustrating another data processing method provided in this application embodiment;

[0054] Figure 8 A flowchart illustrating an example of data processing performed by the first map when there are no aggregation or sorting requirements, as provided in this application embodiment.

[0055] Figure 9 A flowchart illustrating an example of data processing performed by the first map when there is an aggregation requirement but no sorting requirement, as provided in an embodiment of this application.

[0056] Figure 10 A flowchart illustrating an example of data processing performed by the first map when there is a sorting requirement but no aggregation requirement, as provided in this application embodiment.

[0057] Figure 11 A flowchart illustrating an example of data processing performed by the first map when there is an aggregation requirement but no sorting requirement, as provided in an embodiment of this application.

[0058] Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0059] Figure 13 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0060] Figure 14 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0061] Figure 15 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0062] Figure 16 This is a schematic diagram illustrating the connection between computing devices in a computing device cluster, as provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. The terms "system" and "network" in the embodiments of this application can be used interchangeably. "At least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0064] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority or importance of multiple objects.

[0065] Furthermore, the terms "comprising" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but may also include steps or modules not listed.

[0066] Currently, distributed computing frameworks are commonly used to process and analyze massive amounts of data, such as the Hadoop MapReduce (Hadoop MR) framework and Spark. Distributed computing frameworks perform concurrent operations on the data to be processed through multiple computing nodes.

[0067] Please refer to Figure 1 This is a schematic diagram of the architecture of a distributed computing framework. The distributed computing framework includes two mapping nodes, namely mapping node 1 and mapping node 2, and three reduction nodes, namely reduction node 1, reduction node 2 and reduction node 3.

[0068] Mapping nodes 1 and 2, based on the number of reduce nodes and the data processing range of each reduce node, shuffle the data to be processed and store it in different partitions. This data can be, for example, key-value (KV) data obtained by processing each piece of data in the data to be processed. For instance, mapping nodes 1 and 2 store data processed by reduce node 1 in partition 1, data processed by reduce node 2 in partition 2, and data processed by reduce node 3 in partition 3. It should be understood that the storage area corresponding to each mapping node can be a different storage area. Reduce node 1 can retrieve data from partition 1 from the storage areas corresponding to mapping nodes 1 and 2, reduce node 2 can retrieve data from partition 2 from the storage areas corresponding to mapping nodes 1 and 2, and reduce node 3 can retrieve data from partition 3 from the storage areas corresponding to mapping nodes 1 and 2.

[0069] Please refer to Figure 2 This is a schematic diagram of the architecture of a distributed computing framework, which includes a distributed file system (Hadoop Distributed File System, HDFS), four map nodes, and three reduce nodes. The map nodes and reduce nodes can be connected via a network.

[0070] HDFS can store data received from applications by a distributed computing framework (hereinafter referred to as data to be processed), and divide this data into multiple data blocks (or multiple splits) based on the number of map nodes. For example, please refer to... Figure 2This distributed computing framework includes four map nodes. HDFS can divide the data to be processed into four blocks: A1, A2, A3, and A4. These four blocks are then input into the four map nodes respectively. For example, A1 is input into map1, A2 into map2, A3 into map3, and A4 into map4 for processing. It should be understood that map1 through map4 perform the same processing on each data block.

[0071] The map node repartitions and stores the received data blocks based on the corresponding data blocks from the reduce node. For example, please refer to [reference needed]. Figure 2 This distributed computing framework includes three reduce nodes: reduce1, reduce2, and reduce3. Reduce1 corresponds to data block B1, reduce2 to data block B2, and reduce3 to data block B3. Map1 partitions data block A1 based on the data attributes of data blocks B1, B2, and B3, resulting in data B1, B2, and B3 belonging to data blocks B1, B2, and B3 respectively. This data is then stored. Map1 can store data B1 belonging to the same data block (e.g., data block B1) in the same region. Optionally, the map nodes can also process the data block to obtain key-value (KV) data for each piece of data. The resulting KV data is then repartitioned and stored based on the attribute or range of the corresponding key in the reduce nodes. It should be understood that the operations performed on the data blocks by map2 through map4 are the same as those performed by map1, and will not be repeated here.

[0072] The reduce node retrieves and processes the corresponding data from the storage area corresponding to the map node. For example, reduce1 retrieves data belonging to data block B1 from the storage areas corresponding to map1, map2, map3, and map4, processes it, and stores the processed data on HDFS. Optionally, the location where the data blocks processed by the map and reduce nodes are stored may be in a different area of ​​HDFS than the location where the receiving application data is stored. For example, data block A may be stored in HDFS storage area A, and data block B may be stored in HDFS storage area B.

[0073] Understandably, due to memory capacity constraints, after repartitioning its corresponding data blocks, the map node needs to persist the obtained data to the local disk, and the reduce node can retrieve its corresponding data from the local disk of the map node.

[0074] As mentioned earlier, different applications may have data aggregation and / or sorting requirements. Therefore, based on whether aggregation and / or sorting are required during the shuffle operation, the shuffle operation can be divided into four cases. Taking the sort shuffle scheme as an example, the data processing procedures corresponding to these four cases are as follows: Figure 3 As shown.

[0075] The first scenario: No aggregation or sorting requirements.

[0076] The map node caches key-value pairs in an in-memory list and persists them to local storage. Optionally, when persisting the key-value pairs from the in-memory list to local storage, the map node can partition the key-value pairs, for example, storing key-value pairs belonging to the same partition within the same segment of a file. When the reduce node retrieves data from the local storage corresponding to the map node, it can retrieve it from the storage area of ​​the corresponding partition.

[0077] The second scenario: There is an aggregation requirement, but no sorting requirement.

[0078] When a shuffle operation requires aggregation, the aggregation operation can occur in both the map and reduce phases. That is, the map node can perform the aggregation operation, followed by the reduce node. In the following examples, unless otherwise specified, the aggregation operation performed by the map node can also be referred to as a pre-aggregation operation, and the aggregation operation performed by the reduce node can also be referred to as a full aggregation operation.

[0079] When a map node performs a pre-aggregation operation, it can cache key-value pairs in a hash table for pre-aggregation, merging values ​​corresponding to the same key. For example, a map node can iterate through each key-value pair, checking if the key (e.g., key1) already exists in the hash table. If it does, it merges the value (e.g., value1) corresponding to key1 with the value (e.g., value2) corresponding to key1 in the hash table. Optionally, merging can include concatenation, finding the minimum, finding the maximum, or summing operations. The specific implementation can be determined according to the application's requirements. For example, if the application requires summing, then the value corresponding to key1 after merging will be value1 + value2. The map node can persist the key-value pairs from the hash table to local storage. The reduce node can retrieve the data for the corresponding partition from the local storage of the map node and perform a full aggregation operation on the key-value pairs retrieved from different map nodes, caching them in a hash table and merging values ​​containing the same key from different map nodes. It should be understood that the way map nodes persist the key-value pairs of the hash table to local storage is the same as the way map nodes persist the key-value pairs of the memory list to local storage in the first case, and will not be elaborated here.

[0080] The third scenario: There is no aggregation requirement, but there is a sorting requirement.

[0081] For sort shuffle, when the shuffle operation requires sorting, the sorting operation can occur in both the map and reduce phases. That is, the map nodes can perform the sorting operation, and then the reduce nodes can perform the sorting operation. In the following examples, unless otherwise specified, the sorting operation performed by the map nodes can also be called a local sorting operation, and the sorting operation performed by the reduce nodes can also be called a global sorting operation.

[0082] Map nodes cache key-value pairs in an in-memory list, perform local sorting of the key-value pairs by key in the in-memory list, and persist the sorted key-value pairs to local storage. Reduce nodes can retrieve data for the corresponding partition from the local storage of map nodes and perform global sorting of the key-value pairs retrieved from different map nodes by key.

[0083] The fourth scenario: There is a need for aggregation and a need for sorting.

[0084] When the shuffle operation requires aggregation and sorting, pre-aggregation and local sorting operations can be performed on the map nodes, and full aggregation and global sorting operations can be performed on the reduce nodes. The process of pre-aggregation on map nodes can be referenced from the second scenario, as can the process of full aggregation on reduce nodes; similarly, the process of local sorting on map nodes can be referenced from the third scenario, as can the process of global sorting on reduce nodes. These details will not be elaborated upon here. Optionally, map nodes can perform pre-aggregation before local sorting, and reduce nodes can perform full aggregation before global sorting.

[0085] However, in the above technical solutions, when the shuffle operation requires aggregation, the map node performs pre-aggregation operations on all key-value pairs to be processed. Similarly, when the shuffle operation requires sorting, the map node performs local sorting operations on all key-value pairs to be processed. However, application workloads on distributed computing frameworks are diverse. Some applications' data is not suitable for pre-aggregation operations, while others' data is not suitable for local sorting operations, requiring only full aggregation operations or global sorting operations from the reduce nodes. For example, application 1's data requires aggregation, but if the map node performs pre-aggregation operations on it, it may prolong the shuffle operation time and reduce its efficiency.

[0086] In view of this, embodiments of this application provide a data processing method. In this method, if an application requires aggregation or sorting, a map node can first perform aggregation or sorting operations on a portion of the key-value data from the application and determine the duration of these operations. If the duration exceeds a first threshold, it indicates that the application's data is not suitable for aggregation or sorting during the map phase, and the map node can then refrain from performing aggregation or sorting operations on the remaining key-value data to be processed. Furthermore, if the data of a first application requires aggregation and sorting, the map node can first perform aggregation and sorting operations on a portion of the key-value data and determine the duration of these operations. If the duration exceeds a first threshold, it indicates that at least one of the aggregation and sorting operations is time-consuming, and the map node can then refrain from performing the time-consuming operation on the key-value data to be processed. For example, if the aggregation operation is time-consuming, the map node can refrain from performing aggregation on the key-value data to be processed; if the sorting operation is time-consuming, the map node can refrain from performing sorting on the key-value data to be processed; if both aggregation and sorting are time-consuming, the map node can refrain from performing aggregation and sorting on the key-value data to be processed. In this way, map nodes can adaptively adjust whether aggregation and / or sorting operations are needed based on actual operational efficiency, which can improve the execution efficiency of shuffle operations and thus improve the running efficiency of the distributed computing framework.

[0087] Please refer to Figure 4 This is a schematic diagram of the architecture of a distributed processing system provided in an embodiment of this application. The distributed processing system can be applied to a distributed computing frame and includes a driver, an actuator, and a memory.

[0088] The driver is used to transform the application into an execution plan, break down the execution plan into multiple stages, and then divide each stage into one or more tasks according to the number of executors. Each task is assigned to the corresponding executor based on the location of the data, and the storage location of intermediate data generated during the execution of the executor is tracked.

[0089] An executor executes tasks assigned by the driver and stores the resulting intermediate data in a preset location. Optionally, the executor includes an aggregation sorting analyzer and an aggregation sorting regulator. The aggregation sorting analyzer analyzes the effectiveness of pre-aggregation and local sorting in the map phase, such as whether the duration of pre-aggregation and local sorting meets preset conditions. The aggregation sorting regulator adjusts the execution strategy of the executor in the map phase based on the results analyzed by the aggregation sorting analyzer. For example, if the pre-aggregation effect is poor, such as the pre-aggregation duration exceeding a threshold, it indicates that the map phase is not suitable for pre-aggregation operations. The aggregation sorting regulator can adjust the executor's execution strategy, such as not performing pre-aggregation operations on the remaining unprocessed data. Optionally, one executor can correspond to one map node, or one executor can correspond to multiple map nodes. The following embodiment uses one executor corresponding to one map node as an example.

[0090] The storage device is used to store intermediate data generated by the executor. Optionally, the storage device can be a distributed file system, which can improve the reliability of data storage compared to local storage. For example, the distributed file system is configured with two replicas (i.e., data is stored through two storage nodes), so that if one storage node fails, data can be retrieved from the other storage node. It should be understood that the storage device can be deployed on the same server as the executor, or it can be deployed on different servers based on the principle of storage-compute separation. This application does not limit the location of the storage device.

[0091] Optionally, the distributed computing framework may also include a central control regulator, used to determine the execution strategy of the executors based on the data collected by the aggregation analyzer, and send the execution strategy to the aggregation and sorting regulator, which adjusts the execution strategy accordingly. For example, please refer to... Figure 5The central control controller can receive data collected by aggregation sorting analyzer 1 and aggregation sorting analyzer 2, and analyze the effects of pre-aggregation and / or local sorting performed by executors 1 and 2 based on the received data. Based on the analysis results, it determines execution strategy 1 and execution strategy 2 for executors 1 and 2 respectively, and sends execution strategy 1 to aggregation sorting controller 1 and execution strategy 2 to aggregation sorting controller 2. Optionally, execution strategy 1 and execution strategy 2 can be the same or different. For example, if the central control controller determines based on the received data analysis that the pre-aggregation operation performed by executor 1 is more effective than that performed by executor 2, the central control controller can determine that execution strategy 1 for executor 1 is to not perform pre-aggregation on the remaining unprocessed data, while execution strategy 2 for executor 2 is to perform pre-aggregation on the remaining unprocessed data. Alternatively, if the central control controller determines based on the received data analysis that the local sorting operations performed by both executors 1 and 2 are less effective, the central control controller can determine that execution strategy 1 for executor 1 and execution strategy 2 for executor 2 are both to not perform local sorting on the remaining unprocessed data.

[0092] Optionally, the master controller periodically sends execution policies to the aggregation sorting controller, for example, every minute. Optionally, during periods when the master controller does not send execution policies to the aggregation sorting controller (e.g., within one minute after sending the master execution policy), the executor can perform corresponding operations on the data to be processed based on the execution policy last sent by the master controller, or it can analyze the effect of the corresponding operations through the aggregation sorting analyzer. After obtaining the analysis results, the aggregation sorting analyzer can send the analysis results to the aggregation sorting controller, which can determine the executor's execution policy based on the analysis results and determine whether the executor's execution policy needs to be adjusted. Optionally, the master controller can run on a driver or as a separate process.

[0093] Based on the above, the method provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0094] Please refer to Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be performed by... Figure 4 The executor shown can be used, or it can be executed by... Figure 5 The actuator and master controller shown can be used to implement this, or it can also be implemented by... Figure 2 The distributed computing framework shown is implemented using map nodes. The following example uses the method implemented by the first map node.

[0095] S601: The first map performs the first operation on N key-value data from the first application.

[0096] In this embodiment, the first application is the application loaded on the distributed computing framework. When the distributed computing framework receives data from the first application, it can divide the data into multiple data blocks based on the number of map nodes. For a specific division process, please refer to [reference needed]. Figure 2 In HDFS, the process involves partitioning the application's data and sending the resulting data blocks to multiple corresponding map nodes. These map nodes include a first map node, and the data block sent to the first map node is, for example, data block 1.

[0097] When the first map node receives data block 1, it can process the data block 1 to obtain key-value data (KV) corresponding to each data item in data block 1, and then partition and store the obtained KV data according to the data processing range of the reduce node. For example, in a distributed computing framework with two reduce nodes, the first map node can store the KV data belonging to reduce 1 in storage area 1 and the KV data belonging to reduce 2 in storage area 2.

[0098] Optionally, before the first map node stores the key-value pairs belonging to reduce1 in storage area 1, the first map node can also store the key-value pairs belonging to reduce1 in the partition cache area corresponding to reduce1, such as partition cache area 1. When partition cache area 1 is full of data or all the key-value pairs included in data block 1 are written to the partition cache area corresponding to the reduce node (i.e., the map task ends), the data stored in partition cache area 1 is compressed and stored in memory cache area 1. When memory cache area 1 is full of data or the map task ends, the data in memory cache area 1 is persisted to local storage area 1. Similarly, the first map node can store the key-value pairs belonging to reduce2 in the partition cache area corresponding to reduce2, such as partition cache area 2. When partition cache area 2 is full of data or the map task ends, the data stored in partition cache area 2 is compressed and stored in memory cache area 2. When memory cache area 2 is full of data or the map task ends, the data in memory cache area 2 is persisted to local storage area 2. The memory cache is the cache area corresponding to the memory, and the default size of both the partition cache and the memory cache is 64KB. Optionally, the partition cache and memory cache are allocated from memory for the first map when it processes data, and do not need to be reserved in advance. This reduces the number of times the first map accesses local storage, thereby reducing I / O overhead.

[0099] Optionally, when the first map partitions and stores the multiple key-value pairs obtained according to the data processing range corresponding to the reduce nodes, the data processing requirements of the first application can also be determined. Based on these processing requirements, the shuffle operation requirements can be determined, and corresponding operations can be performed on the multiple key-value pairs according to the shuffle operation requirements. The shuffle operation requirements include the following four cases:

[0100] The first scenario: No aggregation or sorting requirements.

[0101] The second scenario: There is an aggregation requirement, but no sorting requirement.

[0102] The third scenario: There is no aggregation requirement, but there is a sorting requirement.

[0103] The fourth scenario: There is a need for aggregation and a need for sorting.

[0104] If the shuffle operation requirement is the first case mentioned above, the first map can skip the aggregation and sorting operations on the multiple key-value pairs to be processed, and directly write each key-value pair to the corresponding partition buffer based on the key.

[0105] If the shuffle operation requirement is not the first case described above, the first map can perform a first operation on N data items, where the first operation includes aggregation and / or sorting operations. For example, if the shuffle operation requirement is the second case described above, i.e., the shuffle operation has aggregation requirements but no sorting requirements, the first map can perform aggregation operations on N key-value pairs; if the shuffle operation requirement is the third case described above, i.e., the shuffle operation has no aggregation requirements but has sorting requirements, the first map can perform sorting operations on N key-value pairs; if the shuffle operation requirement is the fourth case described above, i.e., the shuffle operation has both aggregation and sorting requirements, the first map performs both aggregation and sorting operations on the N key-value pairs. Optionally, the N KVs are the KVs included in data block 1, where the value of N can be determined based on the size of the partition cache, for example, the storage capacity occupied by the N KVs is 64K (i.e., the maximum capacity of the aforementioned partition cache); or, N can be preset; or, N can be determined based on the size of the hash table, for example, the number of KVs obtained by the first map aggregating the N KVs is less than or equal to the threshold of the number of records that the hash table can record.

[0106] S602: The first map determines the first duration of the first operation, and determines the operation performed by the first map on the remaining key-value data from the first application, excluding the N key-value data, based on the first duration.

[0107] As shown in S601, the first operation is an aggregation operation (i.e., the second case in S601), or the first operation is a sorting operation (i.e., the third case in S601), or the first operation is a combination of aggregation and sorting operations (i.e., the fourth case in S601). The first duration of the first operation in each of these three cases will be described below.

[0108] For the second scenario: the first map can obtain the actual time taken to write the N KVs to the first storage area after the aggregation operation (e.g., the second time), where the first storage area is, for example, the partitioned cache area described in S601; and the first map can predict the time taken to write the N KVs to the partitioned cache area without the aggregation operation (e.g., the third time). The first map can determine the difference between the second time and the third time as the duration of the aggregation operation; or, the first map can also determine the second time as the first duration of the aggregation operation.

[0109] Optionally, the time it takes for the first map to predict the N key-value pairs to be written to the partition buffer without aggregation can be calculated based on the time required to write a pre-saved key-value pair to the partition buffer. This pre-saved time can be determined by the first map when writing key-value pairs that do not require aggregation to the partition buffer. For example, if the data for the second application does not require aggregation, the first map can record the actual time taken for each key-value pair to be written to the partition buffer and use the average of the actual times taken for multiple key-value pairs as the total time required for writing a key-value pair to the partition buffer. Alternatively, the pre-saved time can also be determined by other maps when writing key-value pairs that do not require aggregation to the partition buffer.

[0110] For the third case: After writing N key-value pairs into the partitioned buffer, the first map performs a sorting operation on the key-value pairs in the partitioned buffer, records the first timestamp of the start of the sorting operation, and records the second timestamp of the end of the sorting operation. The difference between the second timestamp and the first timestamp is determined as the first duration of the sorting operation.

[0111] For the fourth scenario: the first map can obtain the actual time (e.g., the second duration) after the N KVs are aggregated and sorted, and the second storage area is, for example, the memory cache area described in S601. The first map can also predict the time (e.g., the third duration) after the N KVs are not aggregated and sorted. The first map can determine that the difference between the second duration and the third duration is the duration experienced by the aggregation and sorting operations; or, the first map can also determine the second duration as the first duration experienced by the aggregation and sorting operations.

[0112] Optionally, the third duration predicted by the first map can be determined, for example, based on the time required to write a pre-saved key-value pair to the partition buffer, the time required to compress N key-value pairs, and the time required to store the compressed key-value pairs in the memory buffer. For example, if the time required to write a key-value pair to the partition buffer is A, the time required to compress N key-value pairs is B, and the time required to store the compressed key-value pairs in the memory buffer is C, then the third duration is N×A+B+C. The method for determining the time required to write a pre-saved key-value pair to the partition buffer is the same as in the second case, and will not be repeated here. Furthermore, the time required to compress N key-value pairs and the time required to store the compressed key-value pairs in the memory buffer can also be determined by the first map based on the data processed by the second application, or by other map nodes processing the data of the second application.

[0113] After determining the first duration of the first operation, the first map function can determine the effect of the first operation based on the first duration. For example, if the first duration is less than or equal to a first threshold, it indicates that the effect of performing the first operation on the N key-value pairs is good, and the first map function can execute S604. If the first duration is greater than the first threshold, it indicates that the effect of performing the first operation on the N key-value pairs is poor, and the first map function can execute S603. It should be understood that the corresponding first threshold is also different for different first operations.

[0114] Optionally, when the first operation is an aggregation operation, if the first duration is less than or equal to the first threshold and the first duration is greater than the second threshold, it indicates that although the aggregation operation may add less time to the shuffle operation, there is still a possibility that the I / O overhead is large. Therefore, the first map can also obtain the number M of KVs obtained after the aggregation operation of N KVs. If the ratio of M to N is less than or equal to the third threshold, it indicates that the aggregation operation reduces the number of KVs, which can reduce the I / O overhead. At this time, S604 can be executed. If the ratio of M to N is greater than the third threshold, it indicates that the aggregation operation reduces the number of KVs, and the I / O overhead is still large. Therefore, the first map can execute S603.

[0115] Alternatively, the first map can also determine the first storage capacity occupied by the M KVs obtained after the aggregation operation on the N KVs and the second storage capacity occupied by the N KVs. If the ratio of the first storage capacity to the second storage capacity is less than or equal to the fourth threshold, it indicates that the storage capacity occupied by the KVs after the aggregation operation is small. This can reduce the number of times the partition cache data is stored in the memory cache, thus reducing I / O overhead. At this time, S604 can be executed. If the ratio of the first storage capacity to the second storage capacity is greater than the third threshold, it indicates that the storage capacity occupied by the KVs after the aggregation operation is large, and the I / O overhead is still large. Therefore, the first map can execute S603.

[0116] S603: The first map does not perform the second operation on the remaining key-value data from the first application other than the N key-value data.

[0117] The second operation is related to the first operation. For example, if the first operation is an aggregation operation, then the second operation is an aggregation operation; if the first operation is a sorting operation, then the second operation is a sorting operation; if the first operation is both an aggregation operation and a sorting operation, then the second operation is both an aggregation operation and / or a sorting operation. Optionally, if the first operation is both an aggregation operation and a sorting operation, the first map can also determine, if the duration of the first operation is greater than a first threshold, the first sub-duration of the aggregation operation and the second sub-duration of the sorting operation can be determined separately. If the first sub-duration is greater than a fifth threshold and the second sub-duration is less than or equal to a sixth threshold, then the second operation is an aggregation operation; if the first sub-duration is less than or equal to the fifth threshold and the second sub-duration is greater than the sixth threshold, then the second operation is a sorting operation; if the first sub-duration is greater than the fifth threshold and the second sub-duration is greater than the sixth threshold, then the second operation is both an aggregation operation and a sorting operation. It should be understood that when the first operation is an aggregation operation and a sorting operation, if the second operation is an aggregation operation, the first map will not perform aggregation operations on the remaining KVs of the first application other than the N KVs, but it can perform sorting operations. Similarly, if the second operation is a sorting operation, the first map will not perform sorting operations on the remaining KVs of the first application other than the N KVs, but it can perform aggregation operations. The method by which the first map determines the first sub-duration is the same as the method by which the first map determines the first duration of the aggregation operation in the second case in S602, and the method by which the first map determines the second sub-duration is the same as the method by which the first map determines the first duration of the sorting operation in the third case in S602. These details will not be elaborated further here.

[0118] It should be understood that the first map does not perform a second operation on the remaining KV values ​​from the first application other than the N KV values; that is, the first map directly outputs the remaining KV values ​​from the first application other than the N KV values. Therefore, optionally, S603 can also be replaced by outputting the remaining KV values ​​from the first application other than the N KV values.

[0119] Optionally, when outputting the remaining KV values ​​from the first application besides the N KV values, if the first operation is an aggregation operation, the first map writes the remaining KV values ​​from the first application besides the N KV values ​​into the partition buffer; if the first operation is a sorting operation, the first map directly compresses the data in the partition buffer and stores it in the memory buffer when the partition buffer is full. If the first operation is both an aggregation operation and a sorting operation, the first map can further determine which KV values ​​to output directly based on the effects of the aggregation and sorting operations. For example, if it is determined that the effects of both aggregation and sorting operations are poor, the first map can write the remaining KV values ​​from the first application besides the N KV values ​​into the partition buffer, and when the partition buffer is full, directly compress the data in the partition buffer and store it in the memory buffer, that is, it does not perform aggregation and sorting operations on the remaining KV values ​​from the first application besides the N KV values. If it is determined that the effect of aggregation is poor, while the effect of sorting is good, then the first map... A map operation can write the remaining key-value pairs from the first application (excluding the N key-value pairs) into a partitioned cache. When the partitioned cache is full or the map task ends, the map operation sorts the key-value pairs in the partitioned cache. After the sorting operation, the data in the partitioned cache is compressed and stored in the memory cache. If the aggregation operation is determined to be more effective than the sorting operation, the first map operation can aggregate the remaining key-value pairs from the first application (excluding the N key-value pairs) and write them into the partitioned cache. When the partitioned cache is full or the map task ends, the map operation compresses the data in the partitioned cache and stores it in the corresponding memory cache.

[0120] S604: The first map performs the first operation on the remaining key-value data from the first application, excluding the N key-value data.

[0121] Optionally, if the first operation is a sorting operation, or the first operation is an aggregation operation and a sorting operation, after the first map performs the sorting operation, a first identifier can be added to each of the N sorted key-value pairs to indicate that the key-value pair has been sorted, thereby reducing the probability that the reduce node will perform repeated sorting when reading key-value data from different map nodes.

[0122] In this embodiment, after the first map node stores the data in the local storage area, the reduce node can retrieve the data from the storage areas corresponding to different map nodes and perform corresponding operations according to the four cases in S601. The steps performed by the reduce node for different cases are as follows: Figure 1 or Figure 2 The distributed computing framework shown uses the same method for the reduce nodes to perform operations, so it will not be described again here.

[0123] In the above technical solution, if the application requires aggregation or sorting, the map node can first perform aggregation or sorting operations on a portion of the key-value data from the application and determine the duration of the operation. If the duration exceeds a first threshold, it indicates that the application's data is not suitable for aggregation or sorting operations during the map phase, and the map node can refrain from performing aggregation or sorting operations on its key-value data. Similarly, if the first application's data requires aggregation and sorting, the map node can first perform aggregation and sorting operations on a portion of the key-value data and determine the duration of the aggregation and sorting operations. If the duration exceeds a first threshold, it indicates that at least one of the aggregation and sorting operations is time-consuming, and the map node can refrain from performing the time-consuming operation on its key-value data. For example, if the aggregation operation is time-consuming, the map node can refrain from performing aggregation operations on its key-value data; if the sorting operation is time-consuming, the map node can refrain from performing sorting operations on its key-value data; if both aggregation and sorting are time-consuming, the map node can refrain from performing aggregation and sorting operations on its key-value data. In this way, map nodes can adaptively adjust whether aggregation and / or sorting operations are needed based on actual operational efficiency, which can improve the execution efficiency of shuffle operations and thus improve the running efficiency of the distributed computing framework.

[0124] To better understand the technical solution of this application, the data processing method provided in this application will be explained and described below with reference to specific embodiments.

[0125] Please refer to Figure 7 ,for Figure 6 The illustrated embodiment provides an example flowchart of a data processing method. This data processing method is implemented using a first map as an example.

[0126] When the first map receives a data block to be processed (e.g., data block 1), it processes data block 1 to obtain the key-value pairs (KV) corresponding to each data item in data block 1. Then, based on the requirements of the first application, it selects a method to process the obtained KV pairs (e.g., ...). Figure 7 (The selection path is shown).

[0127] For example, if the first application has neither aggregation nor sorting requirements, the first map can choose the no-aggregation path, directly scatter the multiple KV according to partitions and write them to the corresponding partition cache. When the partition cache is full or the map task ends, the no-sorting path is selected, the data in the partition cache is compressed and stored in the corresponding memory cache. When the memory cache is full or the map task ends, the data in the memory cache is persisted to the local storage area.

[0128] Optionally, after the first map task persists the data from the memory cache to the local storage area, it can also first persist the data from the memory cache to the corresponding persistent cache in local storage. When the persistent cache is full or the map task ends, the data from the persistent cache is then persisted to the local storage area. The reduce node can retrieve the corresponding data from the local storage area and output the retrieved data using a non-aggregated, non-sorted path. For example, please refer to... Figure 8 This is a flowchart illustrating an example of data processing performed by the first map when there are no aggregation or sorting requirements, as provided in this application embodiment.

[0129] S801: The first map node determines whether the map task has ended.

[0130] The completion of the map task indicates that all key-value pairs (KV) of the corresponding data block have been written to the partition cache of the reduce node. For example, if the first map task confirms its completion, it means that all KV pairs of data block 1 (the data block to be processed by the first map task) have been written to the partition cache of the reduce node. Therefore, if the first map task determines that the map task has not yet completed, it executes S802; if the first map task determines that the map task has completed, it executes S808.

[0131] S802: The first map obtains the key and the value corresponding to the key contained in the current data, and obtains the KV1 corresponding to the current data.

[0132] S803: The first map writes the KV1 to the corresponding partition cache i based on the data processing range of the reduce node.

[0133] The value of 'i' is related to the number of reduce nodes. For example, if the number of reduce nodes is 2, the value of 'i' can be 1 or 2. The data processing range of the reduce nodes is divided according to the attribute or range of the key. For example, reduce1 processes data with keys from 1 to 100, and reduce2 processes data with keys from 101 to 1000. If the key corresponding to the key obtained in S802 is 58, the first map writes the key to the partition cache corresponding to reduce1 (for example, the aforementioned partition cache 1); if the key corresponding to the key obtained in S802 is 400, the first map writes the key to the partition cache corresponding to reduce2 (for example, the aforementioned partition cache 2).

[0134] S804: The first map checks whether partition cache i is full of data.

[0135] After the first map writes the KV obtained in S802 into the corresponding partition buffer i, it can also determine whether the partition buffer i is full of data. If it is full of data, the first map executes S805; if it is not full of data, the first map can execute S801.

[0136] S805: The first map compresses the data in partition cache i and stores it in memory cache i.

[0137] S806: The first map determines whether the memory cache i is full of data. If it is full, the first map executes S807; if it is not full, the first map can execute S801.

[0138] S807: The first map persists the data in memory cache area i to local storage area i.

[0139] After the first map persists the data in memory cache area i to local storage area i, S801 can continue to be executed.

[0140] S808: The first map compresses the data in all partition caches and stores it in the corresponding memory cache.

[0141] For example, the first map compresses the data in partition cache 1 and stores it in memory cache 1, and compresses the data in partition cache 2 and stores it in memory cache 2.

[0142] S809: The first map persists all data from the memory cache to the corresponding storage area in local storage.

[0143] For example, the first map persists the data in memory cache 1 to storage area 1 of local storage, and persists the data in memory cache 2 to storage area 2 of local storage.

[0144] If the first application has aggregation requirements but no sorting requirements, the first map can enable the pre-aggregation path. The first map can perform aggregation operations on N key-value pairs through the pre-aggregation path. For example, the first map can cache the N key-value pairs in a hash table through the pre-aggregation path, and merge the values ​​corresponding to the same key to obtain M key-value pairs. The first map will then scatter the M key-value pairs in the hash table according to the partition and write them into the corresponding partition cache area.

[0145] Optionally, the aggregation sorting analyzer can collect the second time interval after the first map performs the aggregation operation on the N KV values ​​and actually writes them to the partition buffer, and the third time interval after the first map directly writes the N KV values ​​to the partition buffer according to the partitions. Based on the second and third time intervals, the analyzer analyzes the effect of the aggregation operation and sends the analysis results to the aggregation sorting regulator. The aggregation sorting regulator determines the effect of the aggregation operation based on the results. If the aggregation operation is effective, the current pre-aggregation path is maintained to perform aggregation operations on the remaining KV values ​​from the first application (excluding the N KV values). If the aggregation operation is ineffective, the aggregation sorting regulator closes the pre-aggregation path and selects a no-aggregation path. The first map can then use this no-aggregation path to directly write the remaining KV values ​​from the first application (excluding the N KV values) to the partition buffer according to the partitions. Optionally, if the aggregation sorting regulator closes the pre-aggregation path, the aggregation sorting analyzer can continue to analyze the effect of the no-aggregation path and determine whether to restart the pre-aggregation path. For example, if the time taken to write N key-value pairs to the partition buffer via the non-pre-aggregation path is greater than the threshold, the pre-aggregation path is restarted; if the time taken to write N key-value pairs to the partition buffer via the non-pre-aggregation path is less than or equal to the threshold, the pre-aggregation path is not restarted.

[0146] When the partition cache is full or the map task ends, the first map node can choose an unsorted path to compress the data in the partition cache and store it in the corresponding memory cache. When the memory cache is full or the map task ends, the data in the memory cache is persisted to local storage. The reduce node can retrieve the corresponding data from the local storage and, choosing an aggregation-based unsorted path, performs full aggregation using a hash table before outputting the data. For example, please refer to... Figure 9 This is a flowchart illustrating an example of data processing performed by the first map when there is an aggregation requirement but no sorting requirement, as provided in this application embodiment.

[0147] S901: The first map node determines whether the map task has been completed.

[0148] The completion of the map task indicates that all key-value pairs (KV) of the corresponding data block have been written to the hash table. For example, if the first map task confirms that it has ended, it means that all KV pairs of data block 1 (the data block to be processed by the first map task) have been written to the hash table. Therefore, if the first map task determines that it has not been completed, it executes S902; if it determines that it has been completed, it executes S913.

[0149] S902: The first map obtains the key and the value corresponding to the key contained in the current data, and obtains the KV1 corresponding to the current data.

[0150] S903: The first map step checks if there exists a KV2 in the hash table that contains the same key as KV1. If it exists, proceed to S904; otherwise, proceed to S905.

[0151] S904: The first map merges the value contained in KV1 with the value contained in KV2 in the hash table.

[0152] S905: The first map adds the KV1 to the hash table.

[0153] For example, the first map can add the KV1 to the i-th row of the hash table, where no data was recorded before the KV1 was added.

[0154] S906: The first map determines whether the hash table has reached the specified threshold.

[0155] The specified threshold is, for example, the maximum number of records in the hash table. When the hash table reaches the specified threshold, it indicates that the hash table is full of data and no new key-value pairs can be added. Therefore, the data in the hash table needs to be written to the partition cache. At this time, the first map can execute S907. If the hash table is not full of data, new key-value pairs can continue to be added. At this time, the first map can execute S901.

[0156] S907: The first map writes the data in the hash table to the corresponding partition cache based on the data processing range of the reduce node.

[0157] The data processing scope of the reduce node is divided according to the key's attributes or key range. For example, reduce1 processes key-value pairs with keys from 1 to 100, while reduce2 processes key-value pairs with keys from 101 to 1000. The first map writes key-value pairs with keys from 1 to 100 in the hash table to the partition cache corresponding to reduce1 (e.g., the aforementioned partition cache 1), and the first map writes key-value pairs with keys from 101 to 1000 in the hash table to the partition cache corresponding to reduce2 (e.g., the aforementioned partition cache 2).

[0158] Optionally, after the first map writes the key-value pairs from the hash table to the corresponding partition cache, it can also clear the key-value pairs from the hash table to make room for writing new key-value pairs.

[0159] S908: The time taken for all key-value pairs in the first map record hash table to be written to the partition cache is used to analyze the effect of the first map on the aggregation operation.

[0160] Optionally, the first map function can use an aggregation sorting analyzer to collect the time it takes for the first map function to write all the key-value pairs in the hash table into the partitioned cache, and analyze the effect of the aggregation operation. Optionally, after obtaining the analysis results, the aggregation sorting analyzer can also send the results to the aggregation sorting regulator. The aggregation sorting regulator determines the first map function's execution strategy for the remaining key-value pairs based on these results, i.e., whether to perform aggregation operations on the remaining key-value pairs. Figure 7 For example, if the aggregation sorting regulator determines that it is not necessary to perform aggregation operations on the remaining KV, the aggregation sorting regulator can close the pre-aggregation path and select no pre-aggregation path.

[0161] S909: The first map function checks whether partition buffer i is full of data. If it is full, the first map function can execute S910; if it is not full, the first map function can execute S901.

[0162] S910: The first map compresses the data in partition cache i and stores it in memory cache i.

[0163] S911: The first map determines whether the memory cache i is full of data. If it is full, the first map executes S912; if it is not full, the first map can execute S901.

[0164] S912: The first map persists the data in the memory cache area i to the local storage area i.

[0165] After the first map persists the data in the memory cache area i to the local storage area i, S901 can continue to be executed.

[0166] S913: The first map writes the data in the hash table to the corresponding partition cache based on the data processing range of the reduce node.

[0167] S914: The first map compresses the data in all partition caches and stores it in the corresponding memory cache.

[0168] S915: The first map persists all data from the memory cache to the corresponding storage area in local storage.

[0169] If the first application has no aggregation requirement but does require sorting, the first map can choose a non-aggregation path, scattering multiple key-value pairs according to partitions and writing them to the corresponding partition caches. When the first map determines that the first partition cache is full of data or that N key-value pairs have been stored in the first partition cache, it enables a local sorting path. The first map can then use this local sorting path to sort the key-value pairs in the partition cache according to the value of K. In the following embodiment, the example of enabling a local sorting path when the first partition cache is full is used for illustration.

[0170] Optionally, the aggregation sorting analyzer can collect the duration of the sorting operation, analyze the sorting effect based on the duration, and send the analysis results to the aggregation sorting regulator. The aggregation sorting regulator determines the effect of the sorting operation performed by the first map based on the analysis results. If the sorting operation is effective, the current local sorting path is maintained to sort the remaining KV from the first application, excluding the N KV. If the sorting operation is ineffective, the aggregation sorting regulator can close the local sorting path and select a no-sorting path. The first map can then compress and store the data in the partition cache into the corresponding memory cache through this no-sorting path.

[0171] The first map node persists the key-value pairs (KV) in the memory cache to local storage when the cache is full or the map task ends. The reduce node can then retrieve the corresponding KV from the local storage and, using a sorted path without aggregation, globally sort the retrieved KV by key before outputting it. For example, please refer to... Figure 10 This is a flowchart illustrating an example of data processing performed by the first map when there is a sorting requirement but no aggregation requirement, as provided in this application embodiment.

[0172] S1001: The first map node determines whether the map task has ended.

[0173] In this context, "map task completed" indicates that all key-value pairs (KV) of the corresponding data block have been written to the partition cache of the reduce node. For example, if the first map operation confirms that the map task has completed, it means that all KV pairs of data block 1 (the data block to be processed by the first map operation) have been written to the partition cache of the reduce node. Therefore, if the first map operation determines that the map task has not been completed, it executes S1002; if the first map operation determines that the map task has been completed, it executes S1009.

[0174] S1002: The first map obtains the key and the value corresponding to the key contained in the current data, and obtains the KV1 corresponding to the current data.

[0175] S1003: The first map writes the KV1 to the corresponding partition cache i based on the data processing range of the reduce node.

[0176] For details regarding this step, please refer to the relevant content in S803, which will not be repeated here.

[0177] S1004: The first map checks whether partition cache i is full of data.

[0178] After the first map writes KV1 to the corresponding partition buffer i, it can also determine whether the partition buffer i is full of data. If it is full of data, the first map executes S1005; if it is not full of data, the first map can execute S1001.

[0179] S1005: The first map sorts the key-value pairs in partition cache i according to the key and obtains the time taken for the sorting operation, which is used to analyze the effect of the sorting operation performed by the first map.

[0180] Optionally, when the first map operation finishes sorting the key-value pairs in partition cache i by key, the aggregation sorting analyzer can collect the time elapsed during the sorting operation, analyze the sorting effect based on this time, and send the analysis results to the aggregation sorting regulator. The aggregation sorting regulator can then determine the execution strategy for the remaining key-value pairs by the first map operation based on these analysis results, i.e., whether the remaining key-value pairs need to be sorted. Figure 7 For example, if the aggregation sorting regulator determines that it is not necessary to sort the remaining key-value pairs, the aggregation sorting regulator can close the local sorting path and select no sorting path.

[0181] S1006: The first map compresses the data in partition cache i and stores it in memory cache i.

[0182] S1007: The first map determines whether the memory cache i is full of data. If it is full, the first map executes S1008; if it is not full, the first map can execute S1001.

[0183] S1008: The first map persists the data in the memory cache area i to the local storage area i.

[0184] After the first map persists the data in memory cache area i to local storage area i, it can continue to execute S1001.

[0185] S1009: The first map compresses the data in all partition caches and stores it in the corresponding memory cache.

[0186] S1010: The first map persists all data from the memory cache to the corresponding storage area in local storage.

[0187] If the first application requires both aggregation and sorting, the first map can enable a pre-aggregation path. The first map can use this pre-aggregation path to perform aggregation operations on N key-value pairs. For example, the first map can use this pre-aggregation path to first cache the N key-value pairs in a hash table, and then merge the values ​​corresponding to the same key to obtain M key-value pairs. The first map will then shuffle the M key-value pairs in the hash table according to partitions and write them into the corresponding partition cache areas.

[0188] Optionally, the aggregation sorting analyzer can analyze the effect of the aggregation operation to determine the execution strategy of the first map for the remaining KV values ​​from the first application besides the N KV values, i.e., whether an aggregation operation is needed. The aggregation sorting analyzer analyzes the effect of the aggregation operation in a way that... Figure 9 In the corresponding embodiment, the aggregation sorting analyzer analyzes the effect of the aggregation operation in the same way, and determines whether the remaining KV from the first application, excluding the N KV, needs to be aggregated in the same way. Figure 9 In the corresponding embodiments, the method for determining whether to perform aggregation operations on the remaining KVs from the first application, excluding the N KVs, is the same, and will not be repeated here.

[0189] The first map operation initiates a local sorting path when the partitioned cache is full or the map task ends, sorting the key-value pairs in the partitioned cache according to the key (K). Optionally, the aggregate sorting analyzer can analyze the effect of the first map operation to determine the first map's execution strategy for the remaining key-value pairs from the first application (excluding the N key-value pairs), i.e., whether sorting is necessary. The aggregate sorting analyzer analyzes the effect of the sorting operation in a way that... Figure 10 In the corresponding embodiment, the aggregation sorting analyzer analyzes the effect of the sorting operation in the same way, and determines whether a sorting operation is needed for the remaining KVs from the first application other than the N KVs. Figure 9 In the corresponding embodiments, the method for determining whether it is necessary to sort the remaining KVs from the first application (excluding the N KVs) is the same, and will not be repeated here.

[0190] The first map node persists the data in the memory cache to local storage when the cache is full or the map task ends. The reduce node can then retrieve the corresponding data from the local storage and, selecting a path with both aggregation and sorting, performs a full aggregation using a hash table followed by a global sort before outputting the data. For example, please refer to... Figure 11This is a flowchart illustrating an example of data processing performed by the first map when there is an aggregation requirement but no sorting requirement, as provided in this application embodiment.

[0191] S1101: The first map node determines whether the map task has been completed.

[0192] The completion of the map task indicates that all key-value pairs (KV) of the corresponding data block have been written to the hash table. For example, if the first map task confirms that it has been completed, it means that all KV pairs of data block 1 (the data block to be processed by the first map task) have been written to the hash table. Therefore, if the first map task determines that it has not been completed, it executes S1102; if it determines that it has been completed, it executes S1114.

[0193] S1102: The first map obtains the key and the value corresponding to the key contained in the current data, and obtains the KV1 corresponding to the current data.

[0194] S1103: The first map checks if there exists a KV2 in the hash table that contains the same key as KV1. If it exists, proceed to S1104; otherwise, proceed to S1105.

[0195] S1104: The first map merges the value contained in KV1 with the value contained in KV2 in the hash table.

[0196] S1105: The first map adds the KV1 to the hash table.

[0197] S1106: The first map checks whether the hash table has reached the specified threshold. If the hash table has reached the specified threshold, then execute S1107; otherwise, execute S1101.

[0198] S1107: The first map writes the data in the hash table to the corresponding partition cache based on the data processing range of the reduce node.

[0199] S1108: The time taken for all the key-value pairs in the hash table of the first map record to be written to the partition cache is used to analyze the effect of the first map on the aggregation operation.

[0200] The embodiments of each step in S1102 to S1108 above can be referred to the embodiments of the corresponding steps in S902 to S908, and will not be repeated here.

[0201] S1109: The first map function checks whether partition buffer i is full of data. If it is full, the first map function can execute S1110; if it is not full, the first map function can execute S1101.

[0202] S1110: The first map sorts the key-value pairs in partition cache i according to the key and obtains the time taken for the sorting operation, which is used to analyze the effect of the sorting operation of the first map.

[0203] S1111: The first map compresses the data in partition cache i and stores it in memory cache i.

[0204] S1112: The first map determines whether the memory cache i is full of data. If it is full, the first map executes S1113; if it is not full, the first map can execute S1101.

[0205] S1113: The first map persists the data in the memory cache area i to the local storage area i.

[0206] After the first map persists the data in the memory cache area i to the local storage area i, it can continue to execute S1101.

[0207] S1114: The first map writes the data in the hash table to the corresponding partition cache based on the data processing range of the reduce node.

[0208] S1115: The first map compresses the data in all partition caches and stores it in the corresponding memory cache.

[0209] S1116: The first map persists all data from the memory cache to the corresponding storage area in local storage.

[0210] In the above technical solution, when the application requires aggregation and / or sorting operations, aggregation and / or sorting operations are first performed on a portion of the data. The effectiveness of these operations is then analyzed. If the effect is good, aggregation and / or sorting operations can continue on the remaining data from the application (excluding the initial portion). If the effect is poor, aggregation and / or sorting operations can be avoided on the remaining data from the application. For example, if the aggregation operation is ineffective, the key-value pairs can be directly written to the partitioned cache; if the sorting operation is ineffective, the key-value pairs in the partitioned cache can be directly compressed and stored in the memory cache. In this way, map nodes can adaptively adjust whether aggregation and / or sorting operations are needed based on actual operational efficiency, improving the execution efficiency of shuffle operations and thus enhancing the overall efficiency of the distributed computing framework.

[0211] Based on the same technical concept as the above-described data processing method, this application also provides a data processing apparatus that can be used to perform the above-described data processing method. Figures 6 to 11The data processing method shown. For example... Figure 12 As shown, the data processing device includes a processing module 1201, which can be implemented by software or by hardware executing the corresponding software. Specifically, in the data processing device, the modules can be connected through a communication path.

[0212] Processing module 1201 is configured to perform a first operation on N key-value data from a first application and determine a first duration of the first operation. If the first duration is greater than a first threshold, a second operation is not performed on the remaining key-value data from the first application other than the N key-value data. Each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword. The first operation is an aggregation operation and / or a sorting operation. The aggregation operation merges values ​​corresponding to the same keyword, and the sorting operation sorts the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1. Specifically, the first operation is the aggregation operation, and the second operation is the aggregation operation; or the first operation is the sorting operation, and the second operation is the sorting operation; or the first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

[0213] In one possible implementation, the first operation is the aggregation operation, and the processing module 1201 is used to determine the first duration of the first operation by: determining the difference between the second duration and the third duration as the first duration, wherein the third duration is the predicted duration for writing the N key-value data into the first storage area; or, determining the second duration as the first duration; wherein the second duration is the actual duration for writing the N key-value data into the first storage area after the aggregation operation.

[0214] In one possible implementation, the first operation is the aggregation operation, and the processing module 1201 is further configured to: determine the number M of key-value data obtained after performing the aggregation operation on the N key-value data when the first duration is less than or equal to the first threshold and greater than the second threshold, where M is a positive integer less than N; if the ratio of M to N is greater than the third threshold, not perform the aggregation operation on the remaining key-value data from the first application other than the N key-value data.

[0215] In one possible implementation, the first operation is the aggregation operation, and the processing module 1201 is further configured to:

[0216] If the first duration is less than or equal to the first threshold and greater than the second threshold, determine the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; if the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, do not perform the aggregation operation on the remaining key-value data from the first application other than the N key-value data, and the second storage capacity is the storage capacity occupied by the N key-value data.

[0217] In one possible implementation, the first operation is the sorting operation, or the first operation is the sorting operation and the aggregation operation, wherein each of the N key-value data carries a first tag after the sorting operation, the first tag being used to indicate that it has been sorted.

[0218] This application embodiment also provides a data processing device, which can have the following functions: Figure 13 The structure shown indicates that the data processing device can be a computer device or a chip or chip system that enables the computer device to implement the above methods.

[0219] like Figure 13 The data processing device shown may include at least one processor 1301, which is coupled to a memory to read and execute instructions in the memory to implement the steps of the data processing method provided in this application embodiment. Optionally, the data processing device may further include a communication interface 1302 for supporting the data processing device in receiving or sending signaling or data. The communication interface 1302 in the data processing device can be used to interact with other electronic devices. The processor 1301 can be used to enable the data processing device to perform actions such as... Figures 6 to 11 The steps in the method shown. Optionally, the data processing device may further include a memory 1303 storing computer instructions. The memory 1303 may be coupled to the processor 1301 and / or the communication interface 1302 to support the processor 1301 in calling the computer instructions in the memory 1303 to implement, for example... Figures 6 to 11 The steps in the method shown; in addition, the memory 1303 can also be used to store data involved in the method embodiments of this application, for example, to store data and instructions necessary for the communication interface 1302 to realize interaction, and / or to store configuration information necessary for the data processing device to execute the method described in the embodiments of this application.

[0220] This application also provides a computing device 1400. For example... Figure 14As shown, the computing device 1400 includes a bus 1401, a processor 1402, a memory 1403, and a communication interface 1404. The processor 1402, the memory 1403, and the communication interface 1404 communicate with each other via the bus 1401. The computing device 1400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1400.

[0221] Bus 1401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 1402 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1402 may include a path for transmitting information between various components of the computing device 1400 (e.g., memory 1403, processor 1402, communication interface 1404).

[0222] Processor 1402 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0223] Memory 1403 may include volatile memory, such as random access memory (RAM). Processor 1402 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0224] The memory 1403 stores executable program code, and the processor 1402 executes the executable program code to implement the functions of the aforementioned processing module 1201, thereby realizing the data processing method provided in this application embodiment. That is, the memory 1403 stores instructions for executing the data processing method provided in this application embodiment.

[0225] The communication interface 1404 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1400 and other devices or communication networks.

[0226] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0227] like Figure 15 As shown, the computing device cluster includes at least one computing device 1400. The memory 1403 of one or more computing devices 1400 in the computing device cluster may store the same instructions for executing a data processing method provided in the embodiments of this application.

[0228] In some possible implementations, the memory 1403 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing a data processing method provided in the embodiments of this application. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing a data processing method provided in the embodiments of this application.

[0229] It is understood that the memory 1403 in different computing devices 1400 within the computing device cluster can store different instructions, each used to execute the functions of the aforementioned processing module 1201. That is, the instructions stored in the memory 1403 of different computing devices 1400 can implement the functions of one or more modules in the processing module 1201.

[0230] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 16 One possible implementation is shown. For example... Figure 16 As shown, the two computing devices 1400A and 1400B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1403 in computing devices 1400A and 1400B stores instructions for performing the functions of processing module 1201.

[0231] It should be understood that Figure 16 The functions of computing device 1400A shown can also be performed by multiple computing devices 1400. Similarly, the functions of computing device 1400B can also be performed by multiple computing devices 1400.

[0232] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 16 The connection method of the computing device cluster is different in that the memory 1403 of one or more computing devices 1400 in the computing device cluster can store the same instructions for executing a data processing method provided in the embodiments of this application.

[0233] In some possible implementations, the memory 1403 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing a data processing method provided in the embodiments of this application. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for executing a data processing method provided in the embodiments of this application.

[0234] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a data processing method provided in this application embodiment.

[0235] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method provided in this application.

[0236] This application also provides a chip, which may include a processor coupled to a memory for reading and executing software programs stored in the memory to perform the methods involved in any possible implementation of the above method embodiments. Here, "coupling" means that two components are directly or indirectly combined with each other, and such combination may be fixed or movable.

[0237] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, implementation can be entirely or partially in the form of computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0238] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a terminal device. Optionally, the processor and storage medium can also be housed in different components within the terminal device.

[0239] These computer instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. A data processing method, characterized by, The method, applied to the first mapping node among multiple mapping nodes in a distributed computing framework, includes: A first operation is performed on N key-value data from a first application, and a first duration is determined for the first operation. Each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword. The first operation is an aggregation operation and / or a sorting operation. The aggregation operation is used to merge values ​​corresponding to the same keyword in the N key-value data. The sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data. N is an integer greater than 1. If the first duration exceeds the first threshold, the second operation is not performed on the remaining key-value data from the first application other than the N key-value data; wherein, The first operation is the aggregation operation, and the second operation is the aggregation operation; or... The first operation is the sorting operation, and the second operation is the sorting operation; or The first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

2. The method of claim 1, wherein, The first operation is the aggregation operation, and determining the first duration of the first operation includes: The difference between the second duration and the third duration is determined to be the first duration, where the third duration is the predicted duration for writing the N key-value data into the first storage area; or, The second duration is determined to be the first duration; The second duration is the actual time taken for the N key-value data to be written into the first storage area after the aggregation operation.

3. The method as described in claim 1 or 2, characterized in that, The first operation is the aggregation operation, and the method further includes: If the first duration is less than or equal to the first threshold and greater than the second threshold, determine the number M of key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; If the ratio of M to N is greater than a third threshold, the aggregation operation is not performed on the remaining key-value data from the first application other than the N key-value data.

4. The method as described in claim 1 or 2, characterized in that, The first operation is the aggregation operation, and the method further includes: If the first duration is less than or equal to the first threshold and greater than the second threshold, determine the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; If the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, the aggregation operation is not performed on the remaining key-value data from the first application other than the N key-value data. The second storage capacity is the storage capacity occupied by the N key-value data.

5. The method as described in claim 1, characterized in that, The first operation is the sorting operation, or the first operation is the sorting operation and the aggregation operation. Each of the N key-value data carries a first tag after the sorting operation, and the first tag is used to indicate that it has been sorted.

6. A data processing apparatus, characterized in that, The device, applied to the first mapping node among multiple mapping nodes in a distributed computing framework, comprises: A processing module is configured to perform a first operation on N key-value data from a first application and determine a first duration of the first operation. If the first duration is greater than a first threshold, a second operation is not performed on the remaining key-value data from the first application other than the N key-value data. Each key-value data in the N key-value data includes a keyword and a value corresponding to the keyword. The first operation is an aggregation operation and / or a sorting operation. The aggregation operation is used to merge values ​​corresponding to the same keyword, and the sorting operation is used to sort the N key-value data according to the keywords included in the N key-value data, where N is an integer greater than 1. Wherein, the first operation is the aggregation operation, and the second operation is the aggregation operation; or, the first operation is the sorting operation, and the second operation is the sorting operation; or the first operation is the aggregation operation and the sorting operation, and the second operation is the aggregation operation and / or the sorting operation.

7. The apparatus as claimed in claim 6, characterized in that, The first operation is the aggregation operation, and the processing module is used to determine the first duration of the first operation in the following manner: The difference between the second duration and the third duration is determined to be the first duration, and the third duration is the predicted duration for writing the N key-value data into the first storage area; or, The second duration is determined to be the first duration; The second duration is the actual time taken for the N key-value data to be written into the first storage area after the aggregation operation.

8. The apparatus as claimed in claim 6 or 7, characterized in that, The first operation is the aggregation operation, and the processing module is further configured to: When the first duration is less than or equal to the first threshold and greater than the second threshold, the number M of key-value data obtained after performing the aggregation operation on the N key-value data is determined, where M is a positive integer less than N; If the ratio of M to N is greater than a third threshold, the aggregation operation is not performed on the remaining key-value data from the first application other than the N key-value data.

9. The apparatus as claimed in claim 6 or 7, characterized in that, The first operation is the aggregation operation, and the processing module is further configured to: If the first duration is less than or equal to the first threshold and greater than the second threshold, determine the first storage capacity occupied by the M key-value data obtained after performing the aggregation operation on the N key-value data, where M is a positive integer less than N; When the ratio of the first storage capacity to the second storage capacity is greater than the fourth threshold, the aggregation operation is not performed on the remaining key-value data from the first application other than the N key-value data. The second storage capacity is the storage capacity occupied by the N key-value data.

10. The apparatus as claimed in claim 6, characterized in that, The first operation is the sorting operation, or the first operation is the sorting operation and the aggregation operation. Each of the N key-value data carries a first tag after the sorting operation, and the first tag is used to indicate that it has been sorted.

11. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 5.

12. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 5.

Images