An optimization method and terminal for processing task data

Abstract

This invention discloses an optimization method and terminal for processing task data. By setting multiple stages on each task node and monitoring the completion status of each stage in each node, when any stage of the task in any node is completed, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to an optimization method and terminal for processing task data. Background Technology

[0002] Modern internet systems contain massive amounts of data, and various AI training programs require the processing of this foundational data. Distributed task scheduling also deals with processing this massive amount of data. Because of the sheer volume of data, a distributed, multi-node computing approach must be adopted, that is, increasing the number of resource nodes to improve the efficiency of task data processing.

[0003] However, currently, for a distributed task, it's impossible to dynamically add task nodes between the start and end of the task. That is, task nodes cannot be dynamically added during task execution; they can only be added after the task concludes and take effect during the next task execution. But current large tasks require considerable processing time. After executing a portion of the task, if it's found to be taking too long, the inability to dynamically add task nodes prevents further improvements in task execution efficiency. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an optimized method and terminal for processing task data, thereby solving the problem of not being able to add nodes during task processing.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] An optimization method for processing task data includes the following steps:

[0007] S1. Divide and process the data for each task node according to the stage;

[0008] S2. Monitor the completion status of each stage in each task node in real time, and retrieve the task node list when any stage is completed.

[0009] S3. Determine whether a task node has been added to the task node list. If yes, proceed to step S4; otherwise, this task node proceeds to the next processing stage.

[0010] S4. Pause this task node until all task nodes have obtained the task node list, and redistribute tasks according to the latest number of task nodes in the task node list.

[0011] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0012] An optimized terminal for processing task data includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0013] S1. Divide and process the data for each task node according to the stage;

[0014] S2. Monitor the completion status of each stage in each task node in real time, and retrieve the task node list when any stage is completed.

[0015] S3. Determine whether a task node has been added to the task node list. If yes, proceed to step S4; otherwise, this task node proceeds to the next processing stage.

[0016] S4. Pause this task node until all task nodes have obtained the task node list, and redistribute tasks according to the latest number of task nodes in the task node list.

[0017] The beneficial effects of this invention are as follows: It provides an optimized method and terminal for processing task data. By setting multiple stages on each task node and monitoring the completion status of each stage in each node, when any stage of the task in any node is completed, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency. Attached Figure Description

[0018] Figure 1 This is a flowchart of an optimization method for processing task data according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of an optimized terminal for processing task data according to an embodiment of the present invention;

[0020] Label Explanation:

[0021] 1. An optimized terminal for processing task data; 2. A memory; 3. A processor. Detailed Implementation

[0022] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] Please refer to Figure 1 An optimization method for processing task data includes the following steps:

[0024] S1. Divide the data processed by each task node into stages;

[0025] S2. Monitor the completion status of each stage in each task node in real time, and retrieve the task node list when any stage is completed.

[0026] S3. Determine whether a task node has been added to the task node list. If yes, proceed to step S4; otherwise, this task node proceeds to the next processing stage.

[0027] S4. Pause this task node until all task nodes have obtained the task node list, and redistribute tasks according to the latest number of task nodes in the task node list.

[0028] As can be seen from the above description, the beneficial effects of the present invention are as follows: by setting multiple stages on each task node and monitoring the completion status of each stage in each node, when any stage of the task in any node is completed, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency.

[0029] In an embodiment of the present invention, step 2 specifically includes the following steps:

[0030] Set up observation nodes to monitor the completion status of each stage in each task node in real time. When the completion of any stage is detected, retrieve the task node list.

[0031] As can be seen from the above description, by using observation nodes independent of the task nodes to observe the completion status of each stage in each task node in real time, the execution dynamics of the task nodes can be obtained, and the task node list can be pulled when any stage of any node is completed, the automation and accuracy of task management are enhanced.

[0032] Preferably, the observation node is deployed using ZooKeeper.

[0033] As is understandable, ZooKeeper is an open-source distributed coordination service primarily used to simplify the development of distributed applications. It provides a range of key functionalities, including configuration management, naming services, and state synchronization, enabling nodes in a distributed system to work collaboratively. Using ZooKeeper as a deployment tool for observation nodes improves system reliability and scalability. The distributed coordination service provided by ZooKeeper enhances data consistency and system stability in multi-node environments.

[0034] In an embodiment of the present invention, step S0 is included before step S1:

[0035] Set a central node, and use the central node to allocate tasks according to the total number of task nodes;

[0036] Step S4 specifically involves:

[0037] Pause this task node until all task nodes have obtained the task node list. The control center node then obtains the remaining data volume of the task and redistributes the remaining data volume according to the task node list.

[0038] As described above, the initial task allocation by the central node ensures a uniform and fair distribution of task processing. When additional task nodes are added, the central node reallocates the remaining data, ensuring a balanced task load while preventing already processed data from being processed again, thus improving the overall system's processing capacity and resource utilization efficiency.

[0039] In an embodiment of the present invention, step S4 is followed by step S5:

[0040] Detect the task processing speed. If the task processing speed is lower than expected, add a new task node and return to step S1.

[0041] As can be seen from the above description, by monitoring the task processing speed and dynamically adding task nodes when the processing speed is lower than expected, this method enhances the system's adaptability to load changes and ensures the timeliness and efficiency of task processing.

[0042] Please refer to Figure 2 An optimized terminal for processing task data includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it completes the steps of an optimized method for processing task data.

[0043] As can be seen from the above description, the beneficial effects of the present invention are as follows: It provides an execution carrier for an optimized method for processing task data. When executing the above method, multiple stages are set on each task node, and the completion status of each stage in each node is monitored. When any stage of the task in any node is completed, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency.

[0044] This invention provides an optimization method and terminal for processing task data, mainly applied to distributed systems during task execution, dynamically adding task nodes to improve processing efficiency. The following is a detailed description with reference to embodiments:

[0045] Please refer to Figure 1 Embodiment 1 of the present invention is as follows:

[0046] An optimization method for processing task data includes the following steps:

[0047] S1. Divide the data processed by each task node into stages;

[0048] S2. Monitor the completion status of each stage in each task node in real time, and retrieve the task node list when any stage is completed.

[0049] S3. Determine whether a task node has been added to the task node list. If yes, proceed to step S4; otherwise, this task node proceeds to the next processing stage.

[0050] S4. Pause this task node until all task nodes have obtained the task node list, and redistribute tasks according to the latest number of task nodes in the task node list.

[0051] In this embodiment, multiple stages are set on each task node, and the completion status of each stage in each node is monitored. When any stage of the task in any node is completed, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency.

[0052] Embodiment 2 of the present invention is as follows:

[0053] Based on Embodiment 1, step 2 specifically includes the following steps:

[0054] Observation nodes are set up to monitor the completion status of each stage in each task node in real time. When the completion of any stage is detected, the task node list is retrieved. The observation nodes are deployed using ZooKeeper. By using observation nodes independent of the task nodes to observe the completion status of each stage in each task node in real time, obtain the execution dynamics of the task nodes, and retrieve the task node list when any stage of any node is completed, the automation and accuracy of task management are enhanced.

[0055] Step S0 is included before step S1:

[0056] A central node is set up, and tasks are allocated according to the total number of task nodes using the central node; the initial task allocation by the central node ensures the even distribution and fairness of task processing.

[0057] Step S4 specifically involves:

[0058] This task node is paused until all task nodes have obtained the task node list. The control center node then obtains the remaining data for the task and redistributes it according to the task node list. As task nodes increase, the central node redistributes the remaining data, ensuring a balanced task load while preventing already processed data from being processed again, thus improving the overall system's processing capacity and resource utilization efficiency.

[0059] Step S4 is followed by step S5:

[0060] The system monitors task processing speed. When the processing speed is lower than expected, a new task node is added, and the process returns to step S1. By monitoring task processing speed and dynamically adding task nodes when the processing speed is lower than expected, this method enhances the system's adaptability to load changes and ensures the timeliness and efficiency of task processing.

[0061] Embodiment 3 of the present invention is as follows:

[0062] Based on Example 1, an optimization method for processing task data is specifically applied as follows:

[0063] In a big data processing center, a distributed system consisting of multiple task nodes is deployed. These task nodes are responsible for processing large-scale data from various data sources.

[0064] The system has a central node that is responsible for the initial allocation of tasks based on the total number of task nodes.

[0065] Meanwhile, ZooKeeper is deployed as an observation node to monitor the status and progress of each task node in real time.

[0066] Step 0: Initial Task Assignment

[0067] The central node received a big data task to process, with a total data volume of 10TB.

[0068] With 10 task nodes currently available, the central node divides the task into 10 equal parts, and each task node is allocated 1TB of data for processing.

[0069] Step 1: Each task node begins processing the data assigned to it, dividing the data into 8 stages for processing, and periodically reporting its progress to the ZooKeeper observation node.

[0070] The observation nodes record the completion status of each stage and provide a real-time updated list of task nodes.

[0071] Step 2: During the processing, the system detects that the processing speed of a certain task node is lower than expected and quickly reports this finding to the central node.

[0072] The central node decides to dynamically add two new task nodes to speed up data processing.

[0073] Step 3: When all task nodes have completed their current stage, pause the current work, and the central node obtains the remaining data of all tasks.

[0074] Based on the new total number of nodes (10 original nodes plus 2 new ones), the central node redistributes the remaining data, and each node now processes a reduced amount of data. The newly added task nodes begin processing the data segments newly allocated to them.

[0075] Step 4: All task nodes continue to execute the new task assignments, and the ZooKeeper observation nodes continue to monitor the progress of each node.

[0076] If any node's performance is detected to be below expectations, the system immediately adjusts the sampling rate to ensure the task is completed on time. Finally, once all data has been processed, the central node aggregates the results and forwards the processed data to downstream systems.

[0077] Embodiment four of the present invention is as follows:

[0078] Please refer to Figure 2 An optimized terminal 1 for processing task data includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, it completes the steps in any of the optimized methods for processing task data in embodiments one to three described above.

[0079] In summary, the present invention provides an optimized method and terminal for processing task data. By setting multiple stages on each task node and monitoring the completion status of each stage in each node, when any stage of the task is completed at any node, the task node list is retrieved. If the number of task nodes changes, the remaining data is redistributed according to the latest number of task nodes. In this way, task nodes are dynamically added and incorporated into the current task processing process during task execution, thereby improving task processing efficiency.

[0080] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An optimization method for processing task data, characterized by: The method comprises the steps of: S1, dividing and processing data processed by each task node according to stages; S2, monitoring completion of each stage in each task node in real time, and pulling a task node list when any stage is completed; S3, judging whether the task node list increases a task node, and if yes, entering step S4; otherwise, the task node enters a next processing stage; S4, pausing the task node until all task nodes acquire the task node list, and re-distributing tasks according to a latest task node quantity of the task node list; The step 2 specifically comprises the step of: setting an observation node, and monitoring completion of each stage in each task node in real time by using the observation node, and pulling a task node list when any stage is detected to be completed; The step S1 further comprises a step S0 before the step S1: setting a center node, and distributing tasks according to a total number of task nodes by using the center node; The step S4 specifically is: pausing the task node until all task nodes acquire the task node list, acquiring a remaining data quantity of tasks by the center node, and re-distributing the remaining data quantity according to the task node list.

2. The optimization method for processing task data according to claim 1, characterized in that: The observation node is deployed by using a zookeeper.

3. The optimization method for processing task data according to claim 1, wherein: The step S4 further comprises a step S5 after the step S4: detecting a task processing speed, adding a task node when the task processing speed is lower than an expectation, and returning to the step S1.

4. An optimized terminal for processing task data, characterized by: A computer program product, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor completes the following steps when executing the computer program: S1, dividing and processing data processed by each task node according to stages; S2, monitoring completion of each stage in each task node in real time, and pulling a task node list when any stage is completed; S3, judging whether the task node list increases a task node, and if yes, entering step S4; otherwise, the task node enters a next processing stage; S4, pausing the task node until all task nodes acquire the task node list, and re-distributing tasks according to a latest task node quantity of the task node list; The step 2 specifically comprises the step of: setting an observation node, and monitoring completion of each stage in each task node in real time by using the observation node, and pulling a task node list when any stage is detected to be completed; The step S1 further comprises a step S0 before the step S1: setting a center node, and distributing tasks according to a total number of task nodes by using the center node; The step S4 specifically is: pausing the task node until all task nodes acquire the task node list, acquiring a remaining data quantity of tasks by the center node, and re-distributing the remaining data quantity according to the task node list.

5. The optimized terminal for processing task data according to claim 4, characterized in that: The observation node is deployed by using a zookeeper.

6. The optimization terminal for processing task data according to claim 4, characterized in that: The step S4 further comprises a step S5 after the step S4: detecting a task processing speed, adding a task node when the task processing speed is lower than an expectation, and returning to the step S1.

Images