A method and system for task data processing

Abstract

The application discloses a kind of task data processing method and system, first result data and its type are obtained after upstream node processing task data ends, center node sets corresponding type label for downstream node after obtaining first result data and its type, and the type label of downstream node is synchronized to upstream node, so upstream node can send specific type first result data to corresponding downstream node. Meanwhile, upstream node is sequentially sent first result data to downstream node, so downstream node can only process specific type data related to task. Compared with the method that upstream node needs to divide data to downstream node in prior art, in the present application, data is not needed to be evenly divided to each downstream node for processing each time, thereby reducing the number of network interactions and network IO occupation of upstream and downstream, and also reducing the amount of data received by center node from downstream node. In this way, the probability of network failure during task processing is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a task data processing method and system. BACKGROUND

[0002] In the current Internet system, with the rapid increase in the number of users, various business data presents a massive data situation, in order to realize the processing of various massive data, the concept of distributed processing task is proposed. That is, there are multiple nodes, a total task is divided into multiple sub-tasks, each node executes these sub-tasks respectively, and finally the result data calculated by each node is returned to the client as the final result data.

[0003] In the usual way, for example, there are 2 nodes processing data upstream, which will send the results evenly to 4 nodes downstream, that is, each node in the upstream will send 1 / 4 of the data to each node in the downstream, and when the sending is completed, the downstream nodes perform the final calculation and send the calculated result data to the center node for aggregation.

[0004] In the above architecture, the following problems exist:

[0005] 1. The number of network interactions between upstream and downstream nodes is too large, increasing the probability of network failure;

[0006] 2. The number of times and the amount of data of the downstream nodes sending the calculation results to the center node also increase. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a task data processing method and system, which can reduce the interaction times between upstream and downstream nodes in the task, and alleviate the network IO occupation, thereby reducing the probability of network failure during task processing.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is:

[0009] A task data processing method applied to a center node, comprising the steps of:

[0010] S1, receiving first result data and its type sent by an upstream node after processing task data ends;

[0011] S2, setting a corresponding type tag for a downstream node according to the type of the first result data, and synchronizing the type tags of the downstream nodes to the upstream node, so that the upstream node sequentially sends the first result data to the downstream node corresponding to the type of the first result data;

[0012] S3, receiving second result data obtained by the downstream node computing the first result data, and performing aggregated computation on the second result data.

[0013] To solve the above technical problems, another technical solution adopted by the present application is:

[0014] A task data processing system comprises a center node, an upstream node and a downstream node.

[0015] The center node receives first result data and its type sent by the upstream node after the upstream node finishes processing task data.

[0016] The center node sets a corresponding type label for the downstream node according to the type of the first result data, and synchronizes the type label of each downstream node to the upstream node, so that the upstream node sends the first result data to the downstream node corresponding to the type of the first result data.

[0017] The center node receives second result data obtained by the downstream node computing the first result data, and performs aggregated computation on the second result data.

[0018] The present application has the beneficial effects that the upstream node obtains first result data and its type after finishing processing task data, the center node sets a corresponding type label for the downstream node after obtaining the first result data and its type, and synchronizes the type label of the downstream node to the upstream node, so that the upstream node can send first result data of a specific type to the corresponding downstream node. At the same time, the upstream node sequentially sends the first result data to the downstream node, so that the downstream node only processes specific type data related to the task. Compared with the method in the prior art in which the upstream node needs to divide the data equally to the downstream nodes, in the present application, the data does not need to be equally divided to each downstream node for processing each time, thereby reducing the number of network interactions between the upstream and downstream nodes and network IO occupation, and also reducing the amount of data received by the center node from the downstream nodes, so as to reduce the probability of network failure during task processing. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a task data processing method according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a task data processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] To explain the technical content, purposes and effects of the present application in detail, the following will be described in conjunction with the embodiments and the accompanying drawings.

[0022] Please refer toFigure 1 This invention provides a method for processing task data, applied to a central node, comprising the following steps:

[0023] S1. Receive the first result data and its type sent by the upstream node after the task data processing is completed;

[0024] S2. Set a corresponding type label for the downstream node according to the type of the first result data, and synchronize the type label of each downstream node to the upstream node, so that the upstream node sequentially sends the first result data to the downstream node corresponding to the type of the first result data;

[0025] S3. Receive the second result data obtained by the downstream node from calculating the first result data, and perform aggregation calculation on the second result data.

[0026] As described above, the beneficial effects of this invention are as follows: After the upstream node finishes processing the task data, it obtains the first result data and its type. The central node, after obtaining the first result data and its type, sets corresponding type labels for the downstream nodes and synchronizes the type labels of the downstream nodes with the upstream node. Therefore, the upstream node can send the first result data of a specific type to the corresponding downstream node. Furthermore, the upstream node sends the first result data to the downstream nodes sequentially, so the downstream nodes only need to process the specific type of data related to the task. Compared to the existing method where the upstream node needs to evenly distribute data to the downstream nodes, this invention does not require evenly distributing data to each downstream node for processing each time, thereby reducing the number of network interactions between upstream and downstream nodes and network I / O usage, and also reducing the amount of data received by the central node from the downstream nodes. This reduces the probability of network failures during task processing.

[0027] Furthermore, step S1 is preceded by:

[0028] Task data is obtained from upstream nodes, the task data is classified, and the set of task data in each category is used as the first result data, and the type of the first result data is obtained.

[0029] As described above, classifying and aggregating task data through upstream nodes facilitates the configuration of downstream nodes based on the first result data obtained from upstream node processing.

[0030] Furthermore, based on the type of the first result data, corresponding type labels are set for downstream nodes, including:

[0031] Based on the type of each first result data, if the number of types of each first result data is less than or equal to the number of downstream nodes, then a corresponding type label is set for multiple downstream nodes in turn, so that the downstream nodes only process the first result data corresponding to the type label.

[0032] As described above, downstream nodes are labeled according to the labels of the first result data only when the number of downstream nodes is greater than or equal to the number of classification types of the first result data, thereby improving the reliability of subsequent upstream and downstream interactions.

[0033] Furthermore, based on the type of each of the first result data, if the number of types of each of the first result data is greater than the number of downstream nodes, then a secondary classification is performed based on the first result data and its type until the number of classification types is less than or equal to the number of downstream nodes.

[0034] As described above, if the number of types of all first result data is greater than the number of downstream nodes, it means that there are too many types of classification at this time. In this case, further classification is used to reduce the number of classification types of task data until the number of classification types is less than or equal to the number of downstream nodes, thereby ensuring the reliability of data processing.

[0035] Furthermore, based on the type of each of the first result data, if the number of types of each of the first result data is greater than the number of downstream nodes, then the downstream nodes are sequentially split into a preset number of child nodes until the total number of all downstream nodes is greater than or equal to the number of types of each of the first result data.

[0036] As described above, if the number of all first result data types is greater than the number of downstream nodes, it means that the number of downstream nodes is small. In this case, the number of nodes that can be labeled is increased by splitting downstream nodes until the number of classification types is less than or equal to the number of downstream nodes, thus ensuring the reliability of data processing.

[0037] Please refer to Figure 2 Another embodiment of the present invention provides a system for processing task data, including a central node, an upstream node, and a downstream node;

[0038] The central node receives the first result data and its type sent by the upstream node after the upstream node has finished processing the task data;

[0039] The central node sets a corresponding type label for the downstream node according to the type of the first result data, and synchronizes the type labels of each downstream node to the upstream node, so that the upstream node sends the first result data to the downstream node corresponding to the type of the first result data;

[0040] The central node receives the second result data obtained by the downstream node from the calculation of the first result data, and performs aggregation calculation on the second result data.

[0041] As described above, the beneficial effects of this invention are as follows: After the upstream node finishes processing the task data, it obtains the first result data and its type. The central node, after obtaining the first result data and its type, sets corresponding type labels for the downstream nodes and synchronizes the type labels of the downstream nodes with the upstream node. Therefore, the upstream node can send the first result data of a specific type to the corresponding downstream node. Furthermore, the upstream node sends the first result data to the downstream nodes sequentially, so the downstream nodes only need to process the specific type of data related to the task. Compared to the existing method where the upstream node needs to evenly distribute data to the downstream nodes, this invention does not require evenly distributing data to each downstream node for processing each time, thereby reducing the number of network interactions between upstream and downstream nodes and network I / O usage, and also reducing the amount of data received by the central node from the downstream nodes. This reduces the probability of network failures during task processing.

[0042] Furthermore, the central node receives the first result data and its type sent by the upstream node after the task data processing is completed, which includes:

[0043] The upstream node obtains task data, classifies the task data, takes the set of task data in each category as the first result data, and obtains the type of the first result data.

[0044] As described above, classifying and aggregating task data through upstream nodes facilitates the configuration of downstream nodes based on the first result data obtained from upstream node processing.

[0045] Furthermore, based on the type of the first result data, corresponding type labels are set for downstream nodes, including:

[0046] Based on the type of each first result data, if the number of types of each first result data is less than or equal to the number of downstream nodes, then a corresponding type label is set for multiple downstream nodes in turn, so that the downstream nodes only process the first result data corresponding to the type label.

[0047] As described above, downstream nodes are labeled according to the labels of the first result data only when the number of downstream nodes is greater than or equal to the number of classification types of the first result data, thereby improving the reliability of subsequent upstream and downstream interactions.

[0048] Furthermore, based on the type of each of the first result data, if the number of types of each of the first result data is greater than the number of downstream nodes, then a secondary classification is performed based on the first result data and its type until the number of classification types is less than or equal to the number of downstream nodes.

[0049] As described above, if the number of types of all first result data is greater than the number of downstream nodes, it means that there are too many types of classification at this time. In this case, further classification is used to reduce the number of classification types of task data until the number of classification types is less than or equal to the number of downstream nodes, thereby ensuring the reliability of data processing.

[0050] Furthermore, based on the type of each of the first result data, if the number of types of each of the first result data is greater than the number of downstream nodes, then the downstream nodes are sequentially split into a preset number of child nodes until the total number of all downstream nodes is greater than or equal to the number of types of each of the first result data.

[0051] As described above, if the number of all first result data types is greater than the number of downstream nodes, it means that the number of downstream nodes is small. In this case, the number of nodes that can be labeled is increased by splitting downstream nodes until the number of classification types is less than or equal to the number of downstream nodes, thus ensuring the reliability of data processing.

[0052] The task data processing method and system described above are suitable for reducing the number of interactions between upstream and downstream nodes in a task and alleviating network I / O overhead, thereby reducing the probability of network failures during task processing. The following specific implementation methods illustrate this:

[0053] Example 1

[0054] Please refer to Figure 1 A method for processing task data, applied to a central node, includes the following steps:

[0055] S1. Receive the first result data and its type sent by the upstream node after the task data processing is completed.

[0056] In this process, the upstream node first obtains task data, classifies the task data, uses the set of task data in each category as the first result data, and obtains the type of the first result data.

[0057] Specifically, in this embodiment there are 2 upstream nodes and 4 downstream nodes. The upstream nodes will obtain a batch of task data from the data source, classify the batch of task data, and form a task data set according to each classification to obtain the first result data and its type.

[0058] In this embodiment, the task data obtained by the upstream node is assumed to contain two data types of type a and two data types of type b. Therefore, the first result data and its type are:

[0059] Type 1: {typea:number}

[0060] Type 2: {typeb:number}

[0061] After the upstream node obtains the first result data, it sends all the first result data to the central node. The central node receives the first result data and its type sent by the upstream node.

[0062] S2. Set a corresponding type label for the downstream node according to the type of the first result data, and synchronize the type label of each downstream node to the upstream node, so that the upstream node sends the first result data to the downstream node corresponding to the type of the first result data.

[0063] Specifically, based on the type of each first result data, if the number of types of each first result data is less than or equal to the number of downstream nodes, then corresponding type labels are set for multiple downstream nodes in turn, so that the downstream nodes only process the first result data corresponding to the type label.

[0064] Specifically, based on the type of each first result data, the central node will know how many types of data there are in total. Then, it sets labels for the data processing types of downstream nodes and synchronously returns the label results of the downstream nodes to the upstream nodes. At this point, the upstream nodes know which types of data need to be forwarded to which downstream nodes.

[0065] In this embodiment, the upstream nodes are a and b, and both a and b contain data of two types: typea and typeb. At this time, the downstream nodes are assigned type labels in turn according to these two types. Specifically, the first downstream node is labeled with the typea label, the second downstream node is labeled with the typeb label, the third downstream node is labeled with the typea label, and the fourth downstream node is labeled with the typeb label. Therefore, the downstream nodes can be recorded as a1, b1, a2, and b2.

[0066] After synchronizing the label results from downstream nodes back to upstream nodes, the upstream nodes prioritize sorting and classifying the first result data. All typea data from node a is forwarded to a1, and all typeb data is forwarded to b1. Similarly, all typea data from node b is forwarded to a2, and all typeb data is forwarded to b2.

[0067] Each upstream node will find the next usable downstream node with the label corresponding to the category type based on its sorting order and classification, and forward the data to that downstream node.

[0068] S3. Receive the second result data obtained by the downstream node from calculating the first result data, and perform aggregation calculation on the second result data.

[0069] Specifically, once all four downstream nodes have completed their calculations, each downstream node will send only one result data to the central node. The central node only needs to perform aggregation calculations on these four result data.

[0070] Example 2

[0071] The main difference between this embodiment and Embodiment 1 is that the method for setting corresponding type labels for downstream nodes is further defined, specifically:

[0072] Based on the type of each first result data, if the number of types of each first result data is greater than the number of downstream nodes, then a secondary classification is performed based on the first result data and its type until the number of classification types is less than or equal to the number of downstream nodes.

[0073] In this embodiment, if the number of types of all first result data is greater than the number of downstream nodes, it means that there are too many types of classification at this time, and some upstream nodes cannot find the corresponding downstream nodes. Therefore, a second classification can be performed on the basis of the first result data and its types. By further classifying, the number of classification types of task data can be reduced until the number of classification types is less than or equal to the number of downstream nodes, thereby ensuring the reliability of data processing.

[0074] Example 3

[0075] The main difference between this embodiment and Embodiment 1 is that the method for setting corresponding type labels for downstream nodes is further defined, specifically:

[0076] Based on the type of each first result data, if the number of types of each first result data is greater than the number of downstream nodes, then the downstream nodes are sequentially split into a preset number of child nodes until the total number of downstream nodes is greater than or equal to the number of types of each first result data.

[0077] In this embodiment, if the number of all first result data types is greater than the number of downstream nodes, it means that the number of downstream nodes is small and some upstream nodes cannot find the corresponding type of downstream nodes. Therefore, the downstream nodes can be split into a preset number of child nodes. By splitting the downstream nodes, the number of nodes that can be marked can be increased until the number of classification types is less than or equal to the number of downstream nodes, so as to ensure the reliability of data processing.

[0078] Example 4

[0079] Please refer to Figure 2 A task data processing system includes a central node, an upstream node, and a downstream node. The central node implements the various steps executed by the central node as the execution subject in any one of the task data processing methods in embodiments one to three. The upstream node implements the various steps executed by the upstream node as the execution subject in any one of the task data processing methods in embodiments one to three. The downstream node implements the various steps executed by the downstream node as the execution subject in any one of the task data processing methods in embodiments one to three.

[0080] In summary, the task data processing method and system provided by this invention optimizes the above-mentioned problems through a data forwarding architecture. The upstream result data is sorted, and then distributed to specific downstream nodes based on the sorted data type. Specific data types are only sent to specific downstream nodes, and the execution steps of downstream tasks are modified to process only data relevant to their own tasks. Once the downstream final node has finished processing its data, it only needs to send its own calculation results to the central node, further reducing the data volume. This approach effectively reduces the number of upstream and downstream network interactions, decreases the probability of network failures, and also reduces the amount of data sent from downstream nodes to the central node, thus reducing network I / O consumption.

[0081] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method of task data processing, characterized by, The application is applied to a center node, comprising steps of: S1, receiving first result data and its type sent by an upstream node after processing task data ends; S2, setting corresponding type tags for downstream nodes according to the type of the first result data, and synchronizing the type tags of the downstream nodes to the upstream node, so that the upstream node sequentially sends the first result data to the downstream nodes corresponding to the type of the first result data; S3, receiving second result data obtained by the downstream nodes calculating the first result data, and performing aggregate calculation on the second result data; Setting corresponding type tags for downstream nodes according to the type of the first result data comprises: According to the type of each first result data, if the number of the type of each first result data is less than or equal to the number of the downstream nodes, corresponding type tags are set for multiple downstream nodes in turn, so that the downstream nodes only process the first result data corresponding to the type tags; According to the type of each first result data, if the number of the type of each first result data is greater than the number of the downstream nodes, secondary classification is performed based on the first result data and its type until the number of classification types is less than or equal to the number of the downstream nodes, or the downstream nodes are sequentially split into a preset number of sub-nodes until the number of all downstream nodes is greater than or equal to the number of the type of each first result data.

2. The method of claim 1, wherein, Before step S1, comprising: The upstream node obtains task data, classifies the task data, takes each set of classified task data as first result data, and obtains the type of the first result data.

3. A system for task data processing, characterized by Comprising a center node, an upstream node and a downstream node; The center node receives first result data and its type sent by the upstream node after processing task data ends; The center node sets corresponding type tags for the downstream nodes according to the type of the first result data, and synchronizes the type tags of the downstream nodes to the upstream node, so that the upstream node sends the first result data to the downstream nodes corresponding to the type of the first result data; The center node receives second result data obtained by the downstream nodes calculating the first result data, and performs aggregate calculation on the second result data; Setting corresponding type tags for downstream nodes according to the type of the first result data comprises: According to the type of each first result data, if the number of the type of each first result data is less than or equal to the number of the downstream nodes, corresponding type tags are set for multiple downstream nodes in turn, so that the downstream nodes only process the first result data corresponding to the type tags; According to the type of each first result data, if the number of types of each first result data is greater than the number of downstream nodes, secondary classification is performed based on the first result data and its type until the number of classification types is less than or equal to the number of downstream nodes or the downstream nodes are sequentially split into a preset number of sub-nodes until the number of all downstream nodes is greater than or equal to the number of types of each first result data.

4. The system for processing task data according to claim 3, wherein, The center node receives the first result data and its type sent after the upstream node finishes processing the task data, and before that includes: The upstream node obtains task data, classifies the task data, takes each set of classified task data as first result data, and obtains the type of the first result data.

Images