A task processing method, device, equipment, computer storage medium and computer program product

Abstract

Embodiments of the present application disclose a task processing method, device and equipment, a computer storage medium and a computer program product, which comprise: performing grouping processing on a to-be-processed training task for model training to obtain multiple groups of training subtasks, and assigning each group of training subtasks a corresponding service node; executing the multiple groups of training subtasks in a parallel processing manner through the multiple service nodes corresponding to each group of training subtasks; stopping execution of a target group of training subtasks in which a target service node is located when a fault occurs in the target service node during execution of the multiple groups of training subtasks; determining a backup service node, and executing unexecuted training subtasks in the target group of training subtasks through the backup service node and other service nodes; the other service nodes are service nodes corresponding to the target group of training subtasks except the target service node, and the problem of a large amount of waste of processor time is solved.

Description

Technical Field

[0001] This application relates to task processing technology in the field of artificial intelligence, and more particularly to a task processing method, apparatus, device, computer storage medium, and computer program product. Background Technology

[0002] The key to generative artificial intelligence lies in the model's parameter size and the number of training samples. However, large-scale parameter sets and a large number of training samples typically require a significant amount of Graphics Processing Units (GPUs) for training. The high computational and network resource load during training makes the task unstable and prone to interruption. To reduce human intervention, current solutions isolate the faulty node, add a backup machine to the schedulable nodes, automatically reschedule the training task, and then load the latest checkpoint to re-enter the training state. However, this significantly wastes GPU time. To address this issue, related technical solutions have emerged that stop all tasks on all nodes in the model training process when a fault occurs, releasing the training process on the faulty node but not on the non-faulty nodes. This allows the non-faulty nodes to avoid reloading the backup (checkpoint) state when the management node restarts the training task after a failure. However, this solution only optimizes rescheduling and initialization time, which is a relatively small proportion of the overall time required, still resulting in significant waste of processor time and excessive resource consumption. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a task processing method, apparatus, device, computer storage medium, and computer program product, which solves the problem of significant waste of processor time when performing model training tasks in related technologies and avoids excessive resource consumption.

[0004] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0005] A task processing method, the method comprising:

[0006] The training tasks to be processed for model training are grouped to obtain multiple training sub-tasks, and each training sub-task is assigned a corresponding service node.

[0007] The training subtasks are executed in parallel by using multiple service nodes corresponding to each training subtask.

[0008] If a target service node fails during the execution of the multiple training sub-tasks, the execution of the target group training sub-tasks containing the target service node will be stopped.

[0009] Identify backup service nodes, and execute the training sub-tasks that were not executed in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, the other service nodes are service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

[0010] In the above scheme, the step of stopping the execution of the target group training subtasks where the target service node is located when a target service node fails during the execution of the multiple sets of training subtasks includes:

[0011] During the execution of the multiple sets of training sub-tasks, the running status of each set of training sub-tasks is detected according to the target training period N;

[0012] If a fault is detected in the target service node during the i-th training period, the service node corresponding to the target group training subtask is controlled to stop working, so as to stop the execution of the target group training subtask.

[0013] In the above scheme, determining a backup service node and executing the unexecuted training sub-tasks in the target group training sub-tasks through the backup service node and other service nodes includes:

[0014] Determine the target number of the backup service nodes from the public service nodes;

[0015] The target service node is obtained by replacing the target service node with the target number of backup service nodes;

[0016] The target service node cluster is used to execute the training subtasks that were not executed in the target group training subtasks.

[0017] In the above scheme, the step of executing the training subtasks that have not been executed in the target group training subtasks through the target service node cluster includes:

[0018] Determine the current configuration information of the model corresponding to the training task to be processed from other group service nodes; wherein, the other group service nodes are the service nodes corresponding to the training sub-tasks other than the target group training sub-task in the multiple groups of training sub-tasks;

[0019] Based on the current state data, the training subtasks that have not been executed in the target group training subtasks are executed through the target service node cluster.

[0020] In the above scheme, the step of executing the unexecuted training subtasks in the target group training subtasks through the target service node cluster based on the current state data includes:

[0021] Starting from the (i+N)th training period, based on the configuration information of the model corresponding to the (i+N)th training period, the first remaining training subtask is executed through the target service node cluster; wherein, the first remaining training subtask is the task in the target group training subtask that has not been executed since the (i+N)th training period.

[0022] In the above scheme, the step of executing the target group training subtask through the target service node cluster based on the current state data includes:

[0023] Starting from the i-th training period, based on the configuration information of the model corresponding to the i-th training period, the second remaining training subtask is executed through the target service node cluster; wherein, the target number is greater than or equal to the target service node number; the second remaining training subtask is the task in the target group training subtask that has not been executed since the i-th period.

[0024] The method in the above scheme further includes:

[0025] If the target number is equal to the target number of service nodes, stop scheduling the other group of service nodes after the other group of training subtasks has been completed;

[0026] Monitor the completion status of the training subtasks of the target service node cluster;

[0027] Once the training subtask of the target service node cluster has been completed, the scheduling of service nodes in the target service node cluster is stopped.

[0028] A task processing device, comprising:

[0029] The allocation unit is used to group the training tasks to be processed for model training into multiple training sub-tasks, and to assign a corresponding service node to each training sub-task.

[0030] The first processing unit is used to execute the multiple sets of training sub-tasks in a parallel processing manner through multiple service nodes corresponding to each set of training sub-tasks.

[0031] The second processing unit is used to stop executing the target group training subtask where the target service node is located if a target service node fails during the execution of the multiple sets of training subtasks.

[0032] A determining unit is configured to determine backup service nodes and execute unexecuted training sub-tasks in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, the other service nodes are service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

[0033] A task processing device includes: a processor, a memory, and a communication bus;

[0034] The communication bus is used to realize the communication connection between the processor and the memory;

[0035] The processor is used to execute the task processing program in the memory to implement the steps of the task processing method described above.

[0036] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the task processing method described above.

[0037] A computer program product comprising a computer program that, when executed by a processor, implements the above-described task processing method.

[0038] The task processing method, apparatus, device, computer storage medium, and computer program product provided in the embodiments of this application can group training tasks to be processed for model training into multiple groups of training sub-tasks, and assign corresponding service nodes to each group of training sub-tasks. Multiple groups of training sub-tasks are executed in parallel by the multiple service nodes corresponding to each group of training sub-tasks. If a target service node fails during the execution of multiple groups of training sub-tasks, the execution of the target group of training sub-tasks containing the target service node is stopped. A backup service node is determined, and the unexecuted training sub-tasks in the target group of training sub-tasks are executed through the backup service node and other service nodes. The other service nodes are the target group of training sub-tasks. The corresponding service nodes, excluding the target service node, allow for the simultaneous parallel processing of training tasks used for model training. When a service node fails, only the training tasks of the group containing the failed service node are stopped, while training tasks in other groups continue normally. Furthermore, a designated backup service node can replace the failed service node and join its group to complete the training task together with other service nodes, instead of halting all tasks during model training as in related technologies. This significantly reduces processor time consumption and solves the problem of excessive processor time wastage during model training in related technologies, avoiding excessive resource consumption. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a task processing method provided for an embodiment of this application;

[0040] Figure 2 A schematic diagram of service node grouping in a task processing method provided for an embodiment of this application;

[0041] Figure 3 A schematic diagram of the structure of a task processing device provided for an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the structure of a task processing device provided for an embodiment of this application. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0044] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0045] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0046] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0047] This application provides a task processing method, which can be applied to a task processing device, as described above. Figure 1 As shown, the method may include the following steps:

[0048] Step 101: Group the training tasks to be processed for model training to obtain multiple training sub-tasks, and assign a corresponding service node to each training sub-task.

[0049] The training task to be processed can be a training model task used to train the desired model. Multiple sets of training subtasks can be obtained by grouping the training tasks to be processed according to actual needs and the availability of service nodes. Furthermore, a corresponding service node can be assigned to each training subtask in each set.

[0050] It should be noted that each training subtask group can include multiple training subtasks, and the number of training subtasks in each group can be the same. Furthermore, each training task in each training subtask group has a corresponding service node.

[0051] In other embodiments of this application, a service node may refer to a worker node deployed in a Kubernetes (K8s) cluster. In one feasible implementation, the entire training task can be run in a distributed manner by 32 worker nodes, each with 8 GPU chips, resulting in a total of 256 GPUs for parallel training. Figure 2 The eight worker nodes shown can form a parallel model group, for example, workers 0-7 in one group, workers 8-15 in another, and so on up to workers 24-31. Of course, Figure 2 The worker nodes are divided in a sequential manner.

[0052] Step 102: Execute multiple training subtasks in parallel using multiple service nodes corresponding to each training subtask.

[0053] In this system, each training subtask in each group has its own corresponding GPU chip. This means that the worker nodes corresponding to each group of training subtasks can process each of their respective groups of training subtasks in parallel. In this way, the training subtasks in all groups are executed at the same time, which can reduce the time required for model training and save the time used by the GPU. This greatly reduces the training cost required for model training and improves the efficiency of model training.

[0054] It should be noted that for each set of training subtasks, each training subtask within each set of training subtasks is also processed simultaneously in parallel.

[0055] Step 103: If a target service node fails during the execution of multiple training subtasks, stop executing the target group training subtasks where the target service node is located.

[0056] Specifically, during the execution of each training subtask group, if a target service node is found to have failed, all training subtasks in the target group to which that target service node belongs will be stopped. It should be noted that when a target service node fails, only all training subtasks in the target group are stopped; the remaining training subtasks in other groups that are not experiencing failures continue to run normally. In other words, when a target service node fails, the remaining service nodes in other groups that are not experiencing failures will not stop their current training subtasks and will continue to execute them normally.

[0057] In other embodiments of this application, the model is tensor and pipeline parallel, and the model is split across different nodes and chips. Therefore, a complete model is maintained within a model parallel group (i.e., each training subtask group). When the target service node fails, stopping all training subtasks in the target group where the target service node is located can ensure the integrity of the model and the accuracy of the finally trained model.

[0058] In one feasible implementation, the faulty target service node can be... Figure 2 If worker node 31 is shown, then all training subtasks in the group from worker node 24 to worker node 31 (worker24-31) will stop executing; however, the training subtasks in worker groups 0-7, worker groups 8-15... up to worker groups 16-23 will still proceed normally.

[0059] Step 104: Determine the backup service node, and execute the training sub-tasks that have not been executed in the target group training sub-tasks through the backup service node and other service nodes.

[0060] Among them, other service nodes are the service nodes other than the target service node in the service nodes corresponding to the target group training subtasks.

[0061] Specifically, backup service nodes can be determined through an elastic public cloud. Furthermore, once the backup service nodes are ready, they can be rejoined with other service nodes in the current training process. This means that the backup service nodes, together with other service nodes, can jointly complete any unexecuted training sub-tasks in the target group's training sub-tasks; thereby ensuring the smooth completion of the pending training sub-tasks and obtaining the final model.

[0062] In other embodiments of this application, step 103 described above can be implemented in the following manner:

[0063] A1. In executing multiple training sub-tasks, the operation status of each training sub-task is checked according to the target training period N.

[0064] The target training period can be a period set according to the actual application scenario and requirements to detect the execution status of the training tasks to be processed. In one feasible implementation, the target training period can refer to a number of training steps preset according to the number of training steps during model training. For example, the target training period can refer to 1000 steps, with each 1000 steps serving as a checkpoint, that is, the execution status of each group of training sub-tasks is checked once every 1000 steps.

[0065] It should be noted that a training monitoring component can be set up; during the parallel execution of multiple training subtasks, the running status of each training subtask can be detected by the training monitoring component according to the target training period N.

[0066] A2. If a fault is detected in the target service node during the i-th training period, control the service node corresponding to the target group training subtask to stop working, so as to stop the execution of the target group training subtask.

[0067] Here, the i-th training cycle can refer to the number of training steps in which a service node failure is detected. Specifically, if all training subtasks shown in 2 are executed from step 1, and every 1000 steps is a checkpoint, the training monitoring component can detect a failure in worker node 31 at step 9500 (i.e., within the training cycle from 9000 to 1000).

[0068] Specifically, if a fault is detected in the target service node, all service nodes corresponding to the target group training subtasks to which the target service node belongs will suspend the execution of their own training subtasks, thus suspending the execution of all training subtasks in the target group.

[0069] In other embodiments of this application, step 104 described above can be implemented in the following manner:

[0070] B1. Determine the target number of backup service nodes from the public service nodes.

[0071] In this context, public service nodes can refer to public servers in a public cloud. Specifically, after a target service node fails, a target number of public servers can be selected from the public servers in the public cloud and used as backup service nodes.

[0072] It should be noted that the target number can be determined based on the actual needs and requirements of the training task. Specifically, the target number can be greater than or equal to the number of service nodes that have failed.

[0073] B2. Replace the target service node with the target number of backup service nodes to obtain the target service node cluster.

[0074] Specifically, the target service node cluster can be obtained by deleting the target service node from the initial service node cluster corresponding to the target group training subtask and adding the newly created backup service node to the service node cluster.

[0075] In one feasible implementation, the backup service node is an elastic (Elastic) worker32 node, such as... Figure 2 As shown, worker node 31 is removed from the worker16-23 group, and Elastic worker node 32 is added to the group, continuing to form a model parallel group (i.e., target service node cluster) with the remaining worker nodes 24-30.

[0076] B3. Execute the training subtasks that have not been executed in the target group training subtasks through the target service node cluster.

[0077] Specifically, after obtaining the target service node cluster, the service nodes in the target service node cluster can be added to parallel training. That is, the target service node cluster can continue to execute the training subtasks that have not yet been executed in the target group's training subtasks. It should be noted that before adding the target service node cluster to parallel training, the current configuration information of the model can be obtained first, and the training subtasks that have not yet been executed in the target group's training subtasks can be continued to be executed based on the current configuration information.

[0078] In other embodiments of this application, the above-mentioned execution of training subtasks not executed in the target group training subtasks through the target service node cluster includes:

[0079] Determine the current configuration information of the model corresponding to the training task to be processed from other group service nodes.

[0080] Among them, the other group service nodes are the service nodes corresponding to the training subtasks other than the target group training subtask in the multiple training subtasks.

[0081] Specifically, the current configuration information for the model can be retrieved from other service nodes that are not experiencing failures. This configuration information may include the model's parameters and the optimizer's status. It should be noted that, for example... Figure 2 The service nodes in the target service node cluster shown can communicate with other groups of service nodes to synchronize model parameters and optimizer states for synchronized training.

[0082] Based on the current state data, execute the training subtasks that have not been executed in the target group training subtasks through the target service node cluster.

[0083] Specifically, the target service node cluster can execute unexecuted training subtasks from the target group's training subtasks based on the latest model parameters and optimizer status, ensuring the accuracy and precision of the final trained model. It should be noted that these unexecuted training subtasks can be executed starting from either the i-th or i+N-th training period.

[0084] In other embodiments of this application, the above-mentioned execution of unexecuted training subtasks in the target group training subtasks based on the current state data through the target service node cluster includes:

[0085] Starting from the (i+N)th training period, based on the configuration information of the model corresponding to the (i+N)th training period, the first remaining training subtask is executed through the target service node cluster.

[0086] Among them, the first remaining training subtask is the task in the target group training subtask that has not been executed since the i+Nth training cycle.

[0087] Specifically, the current parameters of the model and the current state of the optimizer can be obtained at training cycle i+N. Starting from training cycle i+N, based on the current parameters of the model and the current state of the optimizer, the target service node cluster executes the first remaining training subtask. It should be noted that training subtasks between training cycles i and i+N are not re-executed. Instead, the first remaining training subtask is completed based on the latest obtained model parameters and optimizer state to ensure synchronized progress with other groups of service nodes in completing the pending training tasks. This allows all groups of training subtasks to complete simultaneously, avoiding GPU time consumption. Furthermore, in this scheme, calls to all service nodes can be stopped together after all service nodes have completed their respective training subtasks.

[0088] In one feasible implementation, if a faulty target service node is detected at step 9500 (i.e., within the training period of 9000 to 10000), then... Figure 2 As shown, when other service nodes have reached step 11000, the target service node cluster can execute the first remaining training subtask in parallel.

[0089] In other embodiments of this application, the above-mentioned execution of unexecuted training subtasks in the target group training subtasks based on the current state data through the target service node cluster includes:

[0090] Starting from the i-th training period, based on the configuration information of the model corresponding to the i-th training period, the second remaining training subtask is executed through the target service node cluster.

[0091] Among them, the number of targets is greater than or equal to the number of target service nodes; the second remaining training subtask is the task in the target group training subtask that has not been executed since the i-th cycle.

[0092] Specifically, the current parameters of the model and the current state of the optimizer can be obtained in the i-th training cycle. Starting from the i-th training cycle, based on the current parameters of the model and the current state of the optimizer, the target service node cluster can execute the second remaining training subtask. It should be noted that because the number of backup service nodes is greater than or equal to the number of target service nodes at this time, the training subtasks from the i-th training cycle to the (i+N)-th training cycle are re-executed. Therefore, the second remaining training subtask can be completed based on the latest obtained model parameters and optimizer state, which ensures the accuracy of the trained model while avoiding excessive consumption of GPU time.

[0093] In other embodiments of this application, the method may further include:

[0094] If the target number equals the target number of service nodes, stop scheduling service nodes in other groups after the training subtasks of other groups have been completed.

[0095] Monitor the completion status of training subtasks of the target service node cluster;

[0096] Once the training subtask of the target service node cluster has been completed, the scheduling of service nodes in the target service node cluster is stopped.

[0097] Specifically, if the number of backup service nodes equals the number of target service nodes, then once the training monitoring component detects that the training subtasks of other groups of service nodes have been completed, it immediately stops calling those other groups of service nodes, i.e., it terminates those other groups of service nodes. Simultaneously, the training monitoring component continues to monitor the target service node cluster, and once it detects that the training subtasks in the target service node cluster have been completed, it immediately stops calling the service nodes in the target service node cluster, i.e., it terminates the service nodes in the target service node cluster. This avoids wasting service node resources, reduces time consumption on service nodes, and thus lowers costs; at the same time, it also ensures the training efficiency of the model.

[0098] In other embodiments of this application, if the target number is greater than the target number of service nodes, then the calls to all service nodes can be stopped together after all service nodes have completed their respective training sub-tasks.

[0099] It should be noted that, taking 1024 cards as an example, the solution proposed in this application uses elastic standby machines as backup service nodes. Assuming that each 64 cards is a model parallel group, the lost GPU time for retraining is: [T elastic standby machine creation time] * 64 (GPU) + [T model parallel group rejoining training synchronization time + T model parameter synchronization time] * 1024 (GPU). Among them, the elastic standby machine creation time is approximately 0.5 hours, and the sum of T model parallel group rejoining training synchronization time and T model parameter synchronization time is approximately 5 minutes. Therefore, the GPU time lost by the breakpoint-based uninterrupted training solution designed in this application is approximately 118 hours, which can save 14-39 times, or even 16-41 times, compared to solutions in related technologies. In other words, when a large model training encounters a failure, the training of other service nodes will not stop, only the training of the group containing the target service node will stop, while the tasks of the remaining model parallel groups will continue to be trained, thereby increasing the effective training time of all devices and significantly shortening the overall training GPU time.

[0100] Furthermore, the proposed solution does not require the prior preparation of backup service nodes, so the backup service nodes do not have a standby state; with 10% of the backup service nodes in a 1000-card cluster, the GPU time loss caused by the backup service nodes is reduced from more than 10,000 GPU hours to close to 0, which greatly avoids the standby loss of the backup service nodes.

[0101] The task processing method provided in the embodiments of this application can process training tasks for model training in parallel. When a service node fails, only the training tasks of the group containing the failed service node are stopped, while the training tasks of other groups continue to run normally. At the same time, a determined backup service node can be used to replace the failed service node and join the group to complete the training task together with other service nodes, instead of stopping the tasks of all nodes in the model training as in related technologies. This greatly reduces the loss of processor time and solves the problem of a large waste of processor time when performing model training tasks in related technologies, thus avoiding excessive occupation of resources.

[0102] Based on the foregoing embodiments, embodiments of this application provide a task processing device that can be applied to... Figure 1 In the task processing method provided in the corresponding embodiment, refer to Figure 3 As shown, the task processing device 2 may include: an allocation unit 21, a first processing unit 22, a second processing unit 23, and a determination unit 24, wherein:

[0103] The allocation unit 21 is used to group the training tasks to be processed for model training into multiple training sub-tasks, and to allocate a corresponding service node to each training sub-task.

[0104] The first processing unit 22 is used to execute multiple sets of training sub-tasks in parallel processing through multiple service nodes corresponding to each set of training sub-tasks.

[0105] The second processing unit 23 is used to stop executing the target group training subtask where the target service node is located when a target service node fails during the execution of multiple training subtasks.

[0106] The determining unit 24 is used to determine the backup service nodes and execute the training sub-tasks that have not been executed in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, the other service nodes are the service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

[0107] In other embodiments of this application, the second processing unit 23 is further configured to perform the following steps:

[0108] In executing multiple training sub-tasks, the running status of each training sub-task is detected according to the target training period N;

[0109] If a fault is detected in the target service node during the i-th training period, the service node corresponding to the target group training subtask is controlled to stop working, thereby stopping the execution of the target group training subtask.

[0110] In other embodiments of this application, the determining unit 24 is further configured to perform the following steps:

[0111] Determine the target number of backup service nodes from the public service nodes;

[0112] The target service node cluster is obtained by replacing the target service node with the target number of spare service nodes;

[0113] Execute the training subtasks that were not executed in the target group training subtasks through the target service node cluster.

[0114] In other embodiments of this application, the determining unit 24 is further configured to perform the following steps:

[0115] Determine the current configuration information of the model corresponding to the training task to be processed from the service nodes of other groups; where the service nodes of other groups are the service nodes corresponding to the training sub-tasks of other groups besides the target group training sub-task in multiple groups of training sub-tasks;

[0116] Based on the current state data, execute the training subtasks that have not been executed in the target group training subtasks through the target service node cluster.

[0117] In other embodiments of this application, the determining unit 24 is further configured to, starting from the (i+N)th training period, execute the first remaining training subtask through the target service node cluster based on the configuration information corresponding to the model in the (i+N)th training period; wherein, the first remaining training subtask is the task in the target group training subtask that has not been executed since the (i+N)th training period.

[0118] In other embodiments of this application, the determining unit 24 is further configured to, starting from the i-th training period, execute a second remaining training subtask through the target service node cluster based on the configuration information corresponding to the model in the i-th training period; wherein, the number of targets is greater than or equal to the number of target service nodes; and the second remaining training subtask is the task that has not been executed from the i-th period among the target group training subtasks.

[0119] In other embodiments of this application, the second processing unit 23 is further configured to perform the following steps:

[0120] If the target number equals the target number of service nodes, stop scheduling service nodes in other groups after the training subtasks of other groups have been completed.

[0121] Monitor the completion status of training subtasks of the target service node cluster;

[0122] Once the training subtask of the target service node cluster has been completed, the scheduling of service nodes in the target service node cluster is stopped.

[0123] It should be noted that the specific implementation process of the steps performed by each unit in the embodiments of this application can be referred to Figure 1 The implementation process of the task processing method provided in the corresponding embodiment will not be described in detail here.

[0124] The task processing apparatus provided in the embodiments of this application can process training tasks for model training simultaneously and in parallel. When a service node fails, only the training tasks of the group containing the failed service node are stopped, while the training tasks of other groups continue to run normally. At the same time, a determined backup service node can be used to replace the failed service node and join the group to complete the training task together with other service nodes, instead of stopping the tasks of all nodes in the model training as in related technologies. This greatly reduces the loss of processor time and solves the problem of a large waste of processor time when performing model training tasks in related technologies, thus avoiding excessive occupation of resources.

[0125] Based on the foregoing embodiments, embodiments of this application provide a task processing device that can be applied to... Figure 1 In the task processing method provided in the corresponding embodiment, refer to Figure 4As shown, the task processing device 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein:

[0126] Communication bus 33 is used to realize the communication connection between processor 31 and memory 32;

[0127] The processor 31 is used to execute the task processing program in the memory 32 to perform the following steps:

[0128] The training tasks to be processed for model training are grouped to obtain multiple training sub-tasks, and each training sub-task is assigned a corresponding service node.

[0129] Multiple training subtasks are executed in parallel by using multiple service nodes corresponding to each training subtask.

[0130] If a target service node fails during the execution of multiple training subtasks, the execution of the target group training subtasks containing the target service node will be stopped.

[0131] Identify backup service nodes and execute the training sub-tasks that were not executed in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, other service nodes are service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

[0132] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0133] In executing multiple training sub-tasks, the running status of each training sub-task is detected according to the target training period N;

[0134] If a fault is detected in the target service node during the i-th training period, the service node corresponding to the target group training subtask is controlled to stop working, thereby stopping the execution of the target group training subtask.

[0135] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0136] Determine the target number of backup service nodes from the public service nodes;

[0137] The target service node cluster is obtained by replacing the target service node with the target number of spare service nodes;

[0138] Execute the training subtasks that were not executed in the target group training subtasks through the target service node cluster.

[0139] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0140] Determine the current configuration information of the model corresponding to the training task to be processed from the service nodes of other groups; where the service nodes of other groups are the service nodes corresponding to the training sub-tasks of other groups besides the target group training sub-task in multiple groups of training sub-tasks;

[0141] Based on the current state data, execute the training subtasks that have not been executed in the target group training subtasks through the target service node cluster.

[0142] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0143] Starting from the (i+N)th training period, based on the configuration information of the model corresponding to the (i+N)th training period, the first remaining training subtask is executed through the target service node cluster; wherein, the first remaining training subtask is the task in the target group training subtask that has not been executed since the (i+N)th training period.

[0144] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0145] Starting from the i-th training period, based on the configuration information of the model corresponding to the i-th training period, the second remaining training subtask is executed through the target service node cluster; wherein, the number of targets is greater than or equal to the number of target service nodes; the second remaining training subtask is the task in the target group training subtask that has not been executed since the i-th period.

[0146] In other embodiments of this application, the processor 31 is used to run a computer program and may also perform the following steps:

[0147] If the target number equals the target number of service nodes, stop scheduling service nodes in other groups after the training subtasks of other groups have been completed.

[0148] Monitor the completion status of training subtasks of the target service node cluster;

[0149] Once the training subtask of the target service node cluster has been completed, the scheduling of service nodes in the target service node cluster is stopped.

[0150] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 The task processing methods provided in the corresponding embodiments will not be described in detail here.

[0151] The task processing device provided in the embodiments of this application can process training tasks for model training in parallel. When a service node fails, only the training tasks of the group containing the failed service node are stopped, while the training tasks of other groups continue to run normally. At the same time, a determined backup service node can be used to replace the failed service node and join the group to complete the training task together with other service nodes, instead of stopping the tasks of all nodes in the model training as in related technologies. This greatly reduces the loss of processor time and solves the problem of a large waste of processor time when performing model training tasks in related technologies, thus avoiding excessive occupation of resources.

[0152] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement... Figure 1 The steps of the task processing method provided in the corresponding embodiment.

[0153] Based on the foregoing embodiments, embodiments of this application provide a computer program product, including a computer program that can be executed by a processor 31 to perform... Figure 1 The steps of the task processing method provided in the corresponding embodiment.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A task processing method, characterized in that, The method includes: The training tasks to be processed for model training are grouped into multiple training sub-tasks, and each training sub-task is assigned a corresponding service node; wherein, each training sub-task group includes multiple training sub-tasks; and each training sub-task has a corresponding service node. The training subtasks are executed in parallel by using multiple service nodes corresponding to each training subtask. If a target service node fails during the execution of the multiple training sub-tasks, the execution of the target group training sub-tasks containing the target service node will be stopped. Identify backup service nodes, and execute the training sub-tasks that were not executed in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, the other service nodes are service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

2. The method according to claim 1, characterized in that, The step of stopping the execution of the target group training subtasks where the target service node is located when a target service node fails during the execution of the multiple sets of training subtasks includes: During the execution of the multiple sets of training sub-tasks, the running status of each set of training sub-tasks is detected according to the target training period N; If a fault is detected in the target service node during the i-th training period, the service node corresponding to the target group training subtask is controlled to stop working, so as to stop the execution of the target group training subtask.

3. The method according to claim 1, characterized in that, The step of determining backup service nodes and executing the training sub-tasks that were not executed in the target group training sub-tasks through the backup service nodes and other service nodes includes: Determine the target number of the backup service nodes from the public service nodes; The target service node is obtained by replacing the target service node with the target number of backup service nodes; The target service node cluster is used to execute the training subtasks that were not executed in the target group training subtasks.

4. The method according to claim 3, characterized in that, The step of executing the training subtasks that were not executed in the target group training subtasks through the target service node cluster includes: Determine the current configuration information of the model corresponding to the training task to be processed from other group service nodes; wherein, the other group service nodes are the service nodes corresponding to the training sub-tasks other than the target group training sub-task in the multiple groups of training sub-tasks; Based on the current state data, the training subtasks that have not been executed in the target group training subtasks are executed through the target service node cluster.

5. The method according to claim 4, characterized in that, The step of executing the unexecuted training subtasks in the target group training subtasks through the target service node cluster based on the current state data includes: Starting from the (i+N)th training period, based on the configuration information of the model corresponding to the (i+N)th training period, the first remaining training subtask is executed through the target service node cluster; wherein, the first remaining training subtask is the task in the target group training subtask that has not been executed since the (i+N)th training period.

6. The method according to claim 4, characterized in that, The step of executing the unexecuted training subtasks in the target group training subtasks through the target service node cluster based on the current state data includes: Starting from the i-th training period, based on the configuration information of the model corresponding to the i-th training period, the second remaining training subtask is executed through the target service node cluster; wherein, the target number is greater than or equal to the target service node number; the second remaining training subtask is the task in the target group training subtask that has not been executed since the i-th period.

7. The method according to claim 6, characterized in that, The method further includes: If the target number is equal to the target number of service nodes, stop scheduling the other group of service nodes after the other group of training subtasks has been completed; Monitor the completion status of the training subtasks of the target service node cluster; Once the training subtask of the target service node cluster has been completed, the scheduling of service nodes in the target service node cluster is stopped.

8. A task processing device, characterized in that, include: The allocation unit is used to group the training tasks to be processed for model training into multiple groups of training sub-tasks, and to assign a corresponding service node to each group of training sub-tasks; wherein, each group of training sub-tasks includes multiple training sub-tasks; each training sub-task has a corresponding service node. The first processing unit is used to execute the multiple sets of training sub-tasks in a parallel processing manner through multiple service nodes corresponding to each set of training sub-tasks. The second processing unit is used to stop executing the target group training subtask where the target service node is located if a target service node fails during the execution of the multiple sets of training subtasks. A determining unit is configured to determine backup service nodes and execute unexecuted training sub-tasks in the target group training sub-tasks through the backup service nodes and other service nodes; wherein, the other service nodes are service nodes other than the target service node among the service nodes corresponding to the target group training sub-tasks.

9. A task processing device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute a task processing program in the memory to implement the steps of the task processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the task processing method as described in any one of claims 1 to 7.

11. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the task processing method according to any one of claims 1 to 7.

Images