Method and system for executing distributed tasks

Abstract

The invention relates to a method and system for executing a distributed task, and the method comprises the steps: transmitting the snapshot data of the dependence data which each child node depends on and the task state information of the obtained child node to a central node, and when execution interruption occurs, if a fault child node exists, executing the distributed task. If not, the center node obtains available nodes which do not fail and selects any available node to perform interruption recovery according to the task state information and the snapshot data of the failed sub-node, and continuous execution of the distributed task is realized. Therefore, the center node can select any one available node to recover the fault child node, the limitation that in a traditional mode, interruption recovery can only be conducted by depending on other nodes of the same check point as the fault child node is broken through, and even if the situation that dependent data is partially or completely stored in the fault child node occurs, the fault child node can be recovered. Therefore, the success rate of the interrupt recovery is improved, and the execution efficiency of the distributed task is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for executing distributed tasks. Background Art

[0002] Since current tasks require massive data processing, they are generally processed in a distributed task processing manner. Distributed task processing usually uses checkpoints to save data to ensure the continuation of the task. There are multiple subtasks executing concurrently in the checkpoints of distributed task processing. Each subtask needs to upload its own status information to the central node at a regular interval, which is processed uniformly by the central node. Since each node can only pull the saved information of the corresponding checkpoint uploaded by itself, if a node fails and needs to be restored, it can only be restored through other nodes with the same checkpoint. Moreover, if the failed node stores dependent data on which it depends, other nodes may fail to recover during recovery, resulting in the inability of distributed tasks to continue execution, affecting the execution efficiency of distributed tasks. Summary of the invention

[0003] The technical problem to be solved by the present invention is: the present invention provides a method and system for executing distributed tasks to improve the execution efficiency of distributed tasks.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] In a first aspect, the present invention provides a method for executing a distributed task, comprising:

[0006] Obtaining task status information of a subnode existing at each checkpoint in a distributed task, generating snapshot data of dependent data on which each subnode depends, and transmitting the task status information and the snapshot data to a central node;

[0007] Execute the distributed task, and when an execution interruption occurs, determine whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to achieve continued execution of the distributed task.

[0008] The beneficial effect of the present invention is that: while transmitting the task status information of each child node to the central node, the snapshot data of the dependent data that each child node depends on is also transmitted to the central node, that is, the central node saves the relevant data of the nodes of all checkpoints, breaking the barriers between checkpoints, so that when the execution of distributed tasks is interrupted and there is a faulty child node, the central node can select any available node to restore the faulty child node, breaking the limitation of the traditional mode that only other nodes with the same checkpoint as the faulty child node can be relied on for interruption recovery, and because the central node saves the snapshot data of the dependent data that the child node depends on, even if the dependent data is partially or completely stored in the faulty child node, interruption recovery can be performed, thereby improving the success rate of interruption recovery and improving the execution efficiency of distributed tasks.

[0009] Optionally, generating snapshot data of the dependent data on which each child node depends, and transmitting the task status information and the snapshot data to the central node comprises:

[0010] And synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

[0011] According to the above description, by synchronously generating fast data and asynchronously transmitting snapshot data and task status information, the execution blocking of child nodes is reduced and the transmission efficiency is improved.

[0012] Optionally, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery according to the task status information and snapshot data of the failed subnode, including:

[0013] The number of failures of the faulty subnode is obtained. When the number of failures exceeds 1, the central node arbitrarily selects available nodes with the same number as the number of failures as a recovery node group. The available nodes in the recovery node group are randomly selected for interrupt recovery according to the task status information and snapshot data of the faulty subnode.

[0014] According to the above description, the same number of available nodes as the number of faults is generated as the recovery node group to ensure a sufficient number of available nodes, and when the available nodes are recovering from an interruption, the faulty subnodes are randomly selected, which improves the flexibility of interruption recovery.

[0015] Optionally, when performing interruption recovery according to the task status information and snapshot data of the faulty subnode, the available nodes in the recovery node group are randomly selected to perform interruption recovery, including:

[0016] Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, take the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

[0017] According to the above description, when the random selection method is used for interrupt recovery, the available resource information is greater than or equal to the data volume information, that is, the failure of interrupt recovery due to insufficient available resource information is avoided, thereby improving execution efficiency.

[0018] In a second aspect, the present invention provides a system for executing distributed tasks, comprising:

[0019] An acquisition module is used to acquire task status information of a subnode existing at each checkpoint in a distributed task, generate snapshot data of dependent data on which each subnode depends, and transmit the task status information and the snapshot data to a central node;

[0020] The execution module is used to execute the distributed task. When an execution interruption occurs, it is determined whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to realize the continued execution of the distributed task.

[0021] The beneficial effect of the present invention is that: while transmitting the task status information of each child node to the central node, the snapshot data of the dependent data that each child node depends on is also transmitted to the central node, that is, the central node saves the relevant data of the nodes of all checkpoints, breaking the barriers between checkpoints, so that when the execution of distributed tasks is interrupted and there is a faulty child node, the central node can select any available node to restore the faulty child node, breaking the limitation of the traditional mode that only other nodes with the same checkpoint as the faulty child node can be relied on for interruption recovery, and because the central node saves the snapshot data of the dependent data that the child node depends on, even if the dependent data is partially or completely stored in the faulty child node, interruption recovery can be performed, thereby improving the success rate of interruption recovery and improving the execution efficiency of distributed tasks.

[0022] Optionally, the acquisition module is specifically:

[0023] And synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

[0024] According to the above description, by synchronously generating fast data and asynchronously transmitting snapshot data and task status information, the execution blocking of child nodes is reduced and the transmission efficiency is improved.

[0025] Optionally, the execution module includes:

[0026] The interruption recovery module is used to obtain the number of failures of the faulty sub-node. When the number of failures exceeds 1, the central node arbitrarily selects the same number of available nodes as the recovery node group. The available nodes in the recovery node group are randomly selected when performing interruption recovery based on the task status information and snapshot data of the faulty sub-node.

[0027] According to the above description, the same number of available nodes as the number of faults is generated as the recovery node group to ensure a sufficient number of available nodes, and when the available nodes are recovering from an interruption, the faulty subnodes are randomly selected, which improves the flexibility of interruption recovery.

[0028] Optionally, the interrupt recovery module is specifically:

[0029] Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, take the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

[0030] According to the above description, when the random selection method is used for interrupt recovery, the available resource information is greater than or equal to the data volume information, that is, the failure of interrupt recovery due to insufficient available resource information is avoided, thereby improving execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a method for executing distributed tasks provided by this embodiment;

[0032] Figure 2 A schematic diagram of the overall process of a method for executing distributed tasks provided by this embodiment;

[0033] Figure 3 A schematic diagram of the structure of a system for executing distributed tasks provided by this embodiment.

[0034] [Description of Reference Numerals]

[0035] 1. A system for performing distributed tasks;

[0036] 2. Get the module;

[0037] 3. Execution module; 31. Interrupt recovery module. DETAILED DESCRIPTION

[0038] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0039] Embodiment 1

[0040] Please refer to Figure 1 to Figure 2 The present invention provides a method for executing a distributed task, comprising the steps of:

[0041] S1. Obtain the task status information of the child nodes existing at each checkpoint in the distributed task, generate snapshot data of the dependent data on which each child node depends, and transmit the task status information and the snapshot data to the central node;

[0042] In this embodiment, if Figure 2 As shown, at the beginning of the checkpoint, each child node will pause, obtain the task status information of the child node at the current moment, and generate snapshot data of the dependent data that each child node depends on, and transmit the obtained task status information and snapshot data to the central node.

[0043] At this time, generating snapshot data of the dependent data on which each child node depends in step S1 and transmitting the task status information and the snapshot data to the central node includes:

[0044] S11, and synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

[0045] In this embodiment, if Figure 2As shown, in the traditional distributed task processing method, when each child node obtains its own task status information, it needs to synchronously wait for the task status information to be sent to the central node before starting subsequent processing, which affects the processing efficiency. Therefore, in this embodiment, when obtaining the task status information of the child node, snapshot data of the dependent data that the child node depends on is synchronously generated, and the task status information and snapshot data are written into the local cache. The child node can then perform subsequent processing and transmit the task status information and snapshot data in the local cache to the central node through asynchronous transmission. After the transmission is completed, the task status information and snapshot data in the local cache can be cleared immediately, or cleared after a preset time, for example: half an hour, 24 hours, 3 days, etc., which is set according to actual conditions.

[0046] S2. Execute the distributed task. When an execution interruption occurs, determine whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to achieve continued execution of the distributed task.

[0047] In this embodiment, if Figure 2 As shown, when a distributed task is executed and an execution interruption occurs, if there is a faulty sub-node, the central node obtains an available node that has not failed, and randomly selects any available node from the available nodes to perform interrupt recovery on the faulty sub-node to achieve continued execution of the distributed task.

[0048] At this time, the central node in step S2 obtains available nodes that have not failed, and selects any available node to perform interrupt recovery according to the task status information and snapshot data of the failed subnode, including:

[0049] S21. Obtain the number of failures of the faulty sub-node. When the number of failures exceeds 1, the central node arbitrarily selects available nodes with the same number as the number of failures as a recovery node group. The available nodes in the recovery node group are randomly selected for interrupt recovery based on the task status information and snapshot data of the faulty sub-node.

[0050] In this embodiment, if Figure 2 As shown, when the number of faulty sub-nodes exceeds 1, the central node will arbitrarily select the same number of available nodes as the recovery node group as the number of faults, and the available nodes in the recovery node group will randomly select the corresponding faulty sub-nodes for interrupt recovery.

[0051] At this time, the available nodes in the recovery node group in step S21 are randomly selected to perform interruption recovery according to the task status information and snapshot data of the faulty subnode, including:

[0052] S211. Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, and use the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

[0053] In this embodiment, if Figure 2 As shown, the available resource information of the available node is greater than or equal to the data volume information of the faulty subnode as a constraint condition for the available node to randomly select the faulty subnode for interruption recovery, that is, when the available node randomly selects the faulty subnode, the available resource information of the available node must be greater than or equal to the data volume information of the selected faulty subnode.

[0054] Embodiment 2

[0055] Please refer to Figure 3 This embodiment provides a system for executing distributed tasks, including: an acquisition module 2, an execution module 3 and an interruption recovery module 31.

[0056] The acquisition module 2 is used to obtain the task status information of the sub-nodes existing at each checkpoint in the distributed task, generate snapshot data of the dependent data on which each sub-node depends, and transmit the task status information and the snapshot data to the central node;

[0057] The execution module 3 is used to execute the distributed task. When an execution interruption occurs, it is determined whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to realize the continued execution of the distributed task.

[0058] Specifically, the acquisition module 2 is specifically:

[0059] And synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

[0060] Specifically, the execution module 3 includes:

[0061] The interruption recovery module 31 is used to obtain the number of failures of the faulty sub-node. When the number of failures exceeds 1, the central node arbitrarily selects the same number of available nodes as the recovery node group. The available nodes in the recovery node group are randomly selected when performing interruption recovery based on the task status information and snapshot data of the faulty sub-node.

[0062] Specifically, the interrupt recovery module 31 is specifically:

[0063] Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, take the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

[0064] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0065] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0067] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0068] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0069] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0070] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for executing a distributed task, characterized in that: include: Obtaining task status information of a subnode existing at each checkpoint in a distributed task, generating snapshot data of dependent data on which each subnode depends, and transmitting the task status information and the snapshot data to a central node; Execute the distributed task, and when an execution interruption occurs, determine whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to achieve continued execution of the distributed task.

2. A method for executing distributed tasks as claimed in claim 1, characterized in that: The step of generating snapshot data of the dependent data on which each child node depends, and transmitting the task status information and the snapshot data to the central node comprises: And synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

3. A method for executing distributed tasks as claimed in claim 1, characterized in that: The central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery according to the task status information and snapshot data of the failed subnode, including: The number of failures of the faulty subnode is obtained. When the number of failures exceeds 1, the central node arbitrarily selects available nodes with the same number as the number of failures as a recovery node group. The available nodes in the recovery node group are randomly selected for interrupt recovery according to the task status information and snapshot data of the faulty subnode.

4. A method for executing distributed tasks as claimed in claim 3, characterized in that: The available nodes in the recovery node group are randomly selected to perform interruption recovery according to the task status information and snapshot data of the faulty subnode, including: Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, take the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

5. A system for executing distributed tasks, characterized in that: include: An acquisition module is used to acquire task status information of a subnode existing at each checkpoint in a distributed task, generate snapshot data of dependent data on which each subnode depends, and transmit the task status information and the snapshot data to a central node; The execution module is used to execute the distributed task. When an execution interruption occurs, it is determined whether there is a faulty sub-node. If so, the central node obtains available nodes that have not failed, and selects any available node to perform interrupt recovery based on the task status information and snapshot data of the faulty sub-node, so as to realize the continued execution of the distributed task.

6. A system for executing distributed tasks as claimed in claim 5, characterized in that: The acquisition module is specifically: And synchronously generate snapshot data of the dependent data that each child node depends on, write the task status information and the snapshot data into the local cache, and then asynchronously transmit the task status information and snapshot data in the local cache to the central node through an asynchronous thread.

7. A system for executing distributed tasks as claimed in claim 5, characterized in that: The execution module includes: The interruption recovery module is used to obtain the number of failures of the faulty sub-node. When the number of failures exceeds 1, the central node arbitrarily selects the same number of available nodes as the recovery node group. The available nodes in the recovery node group are randomly selected when performing interruption recovery based on the task status information and snapshot data of the faulty sub-node.

8. A system for executing distributed tasks as claimed in claim 7, characterized in that: The interrupt recovery module is specifically: Obtain the available resource information of each available node in the recovery node group and the data volume information of the faulty sub-node, take the available resource information being greater than or equal to the data volume information as a constraint condition, and follow the constraint condition for selection when performing interrupt recovery by random selection.

Images