Task processing method applied to multiple machines and multiple cards and distributed training management system

Abstract

The invention provides a task processing method applied to multiple machines and multiple cards and a distributed training management system, and relates to the technical field of computers, in particular to the field of distributed machine learning. According to the specific implementation scheme, a to-be-processed task is selected from original tasks; determining a to-be-processed task meeting a preset condition by utilizing a restart frequency tag and / or a historical restart task tag of the to-be-processed task; the preset condition is that the number of restarting times of the to-be-processed task is zero; or, the number of restart times of the to-be-processed task is greater than zero and not greater than a preset threshold value, and there is a historical restart task corresponding to the to-be-processed task and failed in training; and adopting a multi-machine multi-card distributed training mode to restart the to-be-processed task meeting the preset condition, and updating the historical restart task label and the restart frequency label of the to-be-processed task meeting the preset condition. According to the scheme, meaningless restarting of the original task can be avoided, so that resource utilization is optimized, and task processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as artificial intelligence, distributed machine learning, and deep learning. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, the scale and complexity of machine learning models have increased, and the number of model parameters has increased dramatically. Traditional single-card or single-machine multi-card training methods have gradually failed to meet the training needs of these models. In order to meet the training needs of these models, multi-machine multi-card distributed training methods have emerged and gradually become the preferred solution for processing large models with large-scale parameters and complex computing structures.

[0003] However, the existing multi-machine and multi-card distributed training methods often have performance defects to varying degrees. Summary of the invention

[0004] The present disclosure provides a task processing method and a distributed training management system applied to multiple machines and multiple cards to solve or alleviate one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a task processing method, comprising:

[0006] Selecting tasks to be processed from original tasks; wherein the tasks to be processed include original tasks that have failed in training and can be processed; and the original tasks include multi-machine and multi-card tasks;

[0007] Determine the pending tasks that meet the preset conditions by using the restart times label of the pending tasks and / or the historical restart task labels; wherein the preset condition is that the restart times of the pending tasks is zero; or, the restart times of the pending tasks is greater than zero and not greater than a preset threshold, and there is a historical restart task with training failure corresponding to the pending tasks; the restart times represent the times of restarting the pending tasks through the distributed training mode of multiple machines and multiple cards;

[0008] A distributed training method with multiple machines and multiple cards is used to restart pending tasks that meet the preset conditions, and to update the historical restart task labels and restart count labels of pending tasks that meet the preset conditions.

[0009] In a second aspect, the present disclosure provides a distributed training management system, including an elastic training module; wherein:

[0010] The elastic training module is used to select tasks to be processed from original tasks; wherein the tasks to be processed include original tasks that have failed in training and can be processed; the original tasks include multi-machine multi-card tasks;

[0011] The elastic training module is further used to determine the pending tasks that meet the preset conditions by using the restart count label of the pending tasks and / or the historical restart task label; wherein the preset condition is that the restart count of the pending tasks is zero; or, the restart count of the pending tasks is greater than zero and not greater than a preset threshold, and there is a historical restart task corresponding to the pending tasks that failed training; the restart count indicates the number of times the pending tasks are restarted through the distributed training method of multiple machines and multiple cards;

[0012] The elastic training module is also used to adopt a distributed training method of multiple machines and multiple cards to restart pending tasks that meet preset conditions, and update the restart count label and historical restart task label of the pending tasks that meet the preset conditions.

[0013] In a third aspect, an electronic device is provided, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0017] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0018] In a fifth aspect, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0019] The beneficial effects of the technical solution provided by the present disclosure include at least:

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0022] Figure 1is an architectural diagram of a distributed training management system 100 according to an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of an implementation of a task processing method 200 according to an embodiment of the present disclosure;

[0024] Figure 3 It is a process diagram according to an existing task processing method;

[0025] Figure 4 is a flow chart of a task processing method implemented according to the present disclosure;

[0026] Figure 5 An architectural diagram of a distributed training management system and a cloud-native infrastructure according to an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of a distributed training management system 600 according to an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of a distributed training management system 700 according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0032] In order to better illustrate the present disclosure, some nouns or terms involved in the embodiments of the present application are first explained:

[0033] (1) Artificial Intelligence (AI): AI is a new technology science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. In recent years, with the rapid development of AI technology, deep learning models have been increasingly used in various industries.

[0034] (2) Multi-machine Multi-GPU: Multi-machine Multi-GPU is used in the fields of deep learning and large-scale computing. Specifically, multi-machine multi-GPU refers to the use of multiple computers (machines) and multiple graphics cards (GPU, Graphics Processing Unit) on each computer to collaborate in computing. This technology can greatly improve computing performance and shorten task processing time, and is particularly suitable for scenarios such as processing large-scale data sets or training complex models.

[0035] (3) Distributed training: Distributed training is a technology that decomposes the training task of a machine learning or deep learning model into multiple subtasks and performs training in parallel on multiple computing nodes (such as CPU, GPU, TPU, etc.). This technology can significantly improve the speed and scalability of training, thereby more efficiently processing large-scale data sets and complex models.

[0036] (4) PyTorch: an open source deep learning framework.

[0037] (5)TensorFlow: an open source deep learning framework.

[0038] (6) Kubernetes: A cloud-native solution and an open source container orchestration system that can be used to automatically deploy, scale, and manage containerized applications.

[0039] (7) Kubeflow: A cloud-native solution for machine learning based on Kubernetes. Kubeflow has cloud-native features, such as elastic scaling, high availability, and DevOps. In addition, Kubeflow is also used to integrate a large number of tools used in machine learning. Cloud-native refers to a software approach to building, deploying, and managing modern applications in a cloud computing environment; DevOps aims to emphasize cooperation, communication, and collaboration between software development and operation and maintenance teams.

[0040] (8) Training-Operator solution: implemented under the Kubeflow framework, it aims to simplify and optimize the process of training and fine-tuning machine learning models on a Kubernetes cluster.

[0041] (9)Arena: An open source component that simplifies the deployment of distributed tasks.

[0042] (10) Volcano: an open source component that enhances the distributed task scheduling mechanism of the cloud-native solution Kubernetes

[0043] (11) Machine allocation: usually refers to the process in which, in a distributed computing environment, due to some reason (such as resource reallocation, node failure recovery, or adjustment of computing resources according to training requirements, the tasks and responsibilities of one or more computing nodes (or "machines") that originally undertook distributed training tasks are transferred to other available computing nodes.

[0044] With the rapid development of artificial intelligence technology, the scale and complexity of machine learning models are increasing, the number of model parameters is rapidly expanding, and the demand for model training is increasing. In order to meet the training needs of these models, multi-machine and multi-card distributed training methods have emerged.

[0045] Currently, in order to implement distributed training of models on multiple machines and multiple cards, most machine learning frameworks (such as PyTorch, TensorFlow, etc.) provide their own distributed training solutions.

[0046] For example, PyTorch provides its own distributed data parallel (DDP) solution.

[0047] These machine learning frameworks that come with distributed training solutions can achieve data parallelism through their own distributed training solutions, thereby achieving the goal of utilizing multiple computers and multiple GPUs to work together, effectively improving the training speed.

[0048] In addition to machine learning frameworks that come with distributed training solutions, there are also cloud-native components similar to kubeflow in the existing technology. These cloud-native components similar to kubeflow usually enhance the deployment and scheduling of distributed training. This is because cloud-native components take full advantage of the advantages of cloud-native architecture, making distributed training more efficient, flexible, and scalable.

[0049] However, in the case of multiple machines and multiple cards, the distributed training solutions that come with machine learning and cloud-native components like kubeflow often have defects to varying degrees.

[0050] For example, in the case of multiple machines and multiple cards, the distributed training solutions that come with machine learning often lack ease of use and functionality in terms of deployment and scheduling; and cloud native components similar to kubeflow, such as the cloud native components provided by kubeflow (i.e., the Training Operator solution), in the case of time-sharing training, will have the defect of not being able to automatically wake up stopped training tasks when machines are allocated, and the inability to implement more common functions such as scheduling task priorities.

[0051] In order to remedy the above-mentioned defects, the embodiments of the present disclosure propose a distributed training management system, which aims to improve the usability and functionality of distributed training solutions in terms of deployment, scheduling and management.

[0052] Figure 1 is an architecture diagram of a distributed training management system 100 according to an embodiment of the present disclosure. The distributed training management system 100 includes a flexible training module 110 and a graphical user interface 120 .

[0053] Among them, the elastic training module 110 can be used to execute the task processing method proposed in the embodiment of the present disclosure, that is, the elastic training module 110 can be used to select a task to be processed from the original task, and use the restart number label and / or historical restart task label of the task to be processed to determine the task to be processed that meets the preset conditions; and, when the task to be processed meets the preset conditions, a distributed training method of multiple machines and multiple cards is adopted to restart the task to be processed, and the restart number label and historical restart task label carried by the task to be processed are updated.

[0054] The graphical user interface 120 may be used to display display information corresponding to each of the multiple original tasks; wherein the display information includes at least one of the following:

[0055] Training results for the original task;

[0056] The training priority of the original task;

[0057] The training log of the original task;

[0058] The user who submitted the original task.

[0059] In one example, the training result of the original task includes training failure and training success;

[0060] The training priority of the original task includes high priority, medium priority, and low priority;

[0061] The training log of the original task includes an offline log and / or an online log;

[0062] The training log is used to record the training data generated during the original task training.

[0063] The disclosed embodiment adopts a graphical user interface (GUI) to display the properties of the original tasks of distributed training and the current running status of each original task in real time, so that the user can quickly, clearly, concisely and holistically understand the current overall training status and the training status of each original task, thereby improving the user's schedulability and overall controllability of training.

[0064] In addition, the graphical user interface 120 is also used to edit the training priority of the original task.

[0065] In one example, the embodiment of the present disclosure can edit the priority of the original task from medium priority to low priority through the graphical user interface 120 , that is, edit the medium priority task to a low priority task.

[0066] Specifically, when editing the priority of any original task, the Volcano field marked in the original task may be determined through the graphical user interface 120, and the priority of the original task may be edited based on the Volcano field marked in the original task.

[0067] Of course, it should be noted that the above examples are only for illustration, and the embodiments of the present disclosure do not limit the specific method of priority editing.

[0068] The disclosed embodiment can reasonably allocate the training order through the priority scheduling function of the original task.

[0069] For example, the disclosed embodiments can schedule the priority of the original tasks so that when graphics card resources are abundant at night, more graphics card resources can be used to train and restart more original tasks; when graphics card resources are tight during the day, a batch of graphics cards need to be allocated to other businesses, so training and restarting some low-priority original tasks are stopped, and they are automatically started according to priority at night, thereby realizing time-sharing elastic distributed training, ensuring the rationality of resource usage sequence, reducing the impact of insufficient resources on training tasks, and ensuring rational resource utilization.

[0070] The above content briefly introduces the elasticity training module 110 and the graphical user interface 120 proposed in the embodiment of the present disclosure. In addition, in order to further illustrate the present disclosure, the following content of the embodiment of the present disclosure will explain in detail the key content of the present disclosure, that is, how the elasticity training module 110 executes the task processing method proposed in the present disclosure.

[0071] Figure 2 is a flowchart of an implementation of a task processing method 200 according to an embodiment of the present disclosure, comprising:

[0072] S210, selecting a task to be processed from the original tasks; wherein the task to be processed includes the original task that failed in training and can be processed; the original task includes a multi-machine multi-card task;

[0073] S220, using the restart count label of the pending task and / or the historical restart task label, determine the pending task that meets the preset condition; wherein the preset condition is that the restart count of the pending task is zero; or, the restart count of the pending task is greater than zero and not greater than a preset threshold, and there is a historical restart task corresponding to the pending task that failed training; the restart count indicates the number of times the pending task is restarted through a distributed training method of multiple machines and multiple cards;

[0074] S230, using a distributed training method of multiple machines and multiple cards, restarting pending tasks that meet preset conditions, and updating the restart count labels and historical restart task labels of pending tasks that meet preset conditions.

[0075] Compared with the prior art, the embodiments of the present disclosure can reduce the training of unnecessary tasks to be processed by screening the tasks to be processed that meet preset conditions for subsequent processing, thereby optimizing the execution efficiency of the tasks to be processed, helping to speed up training and save computing resources, and improving the overall performance of the system.

[0076] In one example, the restart count label is used to indicate the restart count of the pending task;

[0077] The historical restart task tag is used to indicate the name data of the historical restart task corresponding to the task to be processed.

[0078] Specifically, the embodiment of the present disclosure may use restarts to represent the restart count label; and use fault-tolerant-jobname to represent the historical restart task label.

[0079] For example, if restarts=i corresponding to the pending task, it means that the number of restarts of the pending task is i, where i can be any positive integer, such as 3, 5, 7, etc.

[0080] In one example, if the restart count label corresponding to the pending task is restarts=0, then the restart count of the pending task is zero, that is, the pending task has not been restarted; or, if the restart count label corresponding to the pending task is restarts=2, then the restart count of the pending task is 2, that is, the pending task has been restarted 2 times.

[0081] If the pending task carries a historical restart task tag, and the historical restart task tag is fault-tolerant-jobname=A i , then the name data of the historical restart task corresponding to the pending task is A i , that is, the name of the historical restart task corresponding to the pending task is A iOf course, if the pending task does not carry the historical restart task tag, it means that there is currently no historical restart task corresponding to the pending task, that is, the pending task has not been restarted, that is, the number of restarts of the pending task is zero.

[0082] Typically, after training on multiple original tasks, existing techniques are trained by Figure 3 The method shown performs polling and scanning on multiple original tasks, and restarts the original tasks that fail to be trained.

[0083] However, if Figure 3 The existing methods shown often have the problem of system inefficiency and resource waste caused by erroneously re-starting failed tasks.

[0084] Specifically, Figure 3 As shown, the first polling and the second polling are taken as examples.

[0085] After the training is completed, the existing task processing method usually performs a first poll on all the original tasks (i.e., traverses all the original tasks) to find the original tasks that failed in training; then performs a restart operation on the original tasks that failed in training to obtain the restart task corresponding to the original tasks that failed in training.

[0086] For example, Figure 3 As shown, if the original task that fails in training is original task A, then the existing task processing method usually performs a restart operation on the original task A to obtain the restart task corresponding to the original task A, that is, restart task A1.

[0087] Afterwards, after the first polling is completed, the existing task processing method cannot avoid the interference caused by the original task that has been polled for the first time on this polling, and will still perform a second poll on all tasks (that is, all original tasks and restarted tasks) (that is, traverse all original tasks and restarted tasks) to find out the tasks that failed training, and perform a restart operation on the tasks that failed training to obtain the new restarted task corresponding to the tasks that failed training.

[0088] Since the existing task processing method will not only find the original task of training failure (i.e. the failed task found in the first polling) in the second polling, but also find the restart task of training failure based on the original task of training failure. Therefore, in the second polling of the existing task processing method, the restart task pulled up based on training failure often has duplication.

[0089] For example, Figure 3As shown, after the first polling is completed, if the restart task A1 fails to be trained, then in the second polling, the existing task processing method will find out the original task A that failed to be trained and the restart task A1 that failed to be trained, and perform restart operations on both the original task A and the restart task A1 that failed to be trained, so as to obtain the new restart task corresponding to the original task A, i.e., the restart task A2, and the restart task corresponding to the restart task A1, i.e., the restart task A 12 , then there will be restart task A2 and restart task A 12 Repeated situation.

[0090] In order to avoid unnecessary repetition of tasks due to errors, thereby causing inefficiency and waste of resources in the system, the embodiment of the present disclosure proposes a new task processing method.

[0091] Figure 4 is a flow chart of a task processing method implemented according to the present disclosure, such as Figure 4 As shown, after training multiple original tasks, the embodiment of the present disclosure can be performed as follows Figure 4 The method shown starts a polling for multiple primitive tasks.

[0092] Specifically, Figure 4 The task processing method of the embodiment of the present disclosure includes:

[0093] Step 1: Polling starts, and tasks to be processed are selected from multiple original tasks.

[0094] The disclosed embodiment may determine original tasks that have failed in training and can be processed as tasks to be processed.

[0095] Specifically, the method of selecting a task to be processed from an original task proposed in the embodiment of the present disclosure includes:

[0096] Determine the processability label and training result of the original task; wherein the processability label is used to indicate the processability of the original task; processability includes processability and unprocessability; and the training result includes training success and training failure;

[0097] By using the processability labels and training results of the original tasks, original tasks that failed in training and are processable are selected from the original tasks as tasks to be processed.

[0098] In one example, if the processability of the task to be processed is represented by a processability label (i.e., fault-tolerant), and fault-tolerant=on indicates that the task to be processed is processable, and fault-tolerant=off indicates that the task to be processed is not processable; and the training result of the task to be processed is represented by status, and status=failed indicates that the training result of the task to be processed is a training failure, and status=succeed indicates that the training result of the task to be processed is a training success, then the embodiment of the present disclosure can determine the original task carrying the fault-tolerant=on label and status=failed among multiple original tasks as the task to be processed.

[0099] The method of selecting tasks to be processed through processability labels and training results proposed in the embodiment of the present disclosure can further increase the accuracy of the selected tasks to be processed, accelerate the screening of tasks to be processed, avoid repeated polling of original tasks and historical restart tasks, achieve efficient utilization of the system and avoid waste of key resources.

[0100] Step 2: Use the restart count label of the task to be processed and / or the historical restart task label to determine whether the processing task meets the preset conditions.

[0101] Specifically, the restart count label and / or the historical restart task label of the pending task are used to determine the pending task that meets the preset conditions, including:

[0102] Determine the restart times of the pending task by using the historical restart task tag and / or the restart times tag of the pending task;

[0103] When the number of restarts of the pending task is zero, determining that the pending task meets the preset condition; or,

[0104] When the restart count of the pending task is not zero, the restart count of the pending task and the historical restart task label are used to select a pending task that meets a preset condition from the pending tasks.

[0105] For pending tasks that do not carry a historical restart task tag, the disclosed embodiment can determine that there is currently no historical restart task corresponding to the pending task, that is, the pending task that does not carry a historical restart task tag has never been restarted, that is, the number of restarts of the pending task that does not carry a historical restart task tag is zero, then the disclosed embodiment can directly determine that the pending task that does not carry a historical restart task tag meets the preset conditions.

[0106] For example, take the task to be processed as task B, such as Figure 4As shown, the embodiment of the present disclosure can adopt step 2.1 to determine whether the task B meets the preset conditions, that is, to determine whether the task B carries a historical restart task tag (that is, fault-tolerant-jobname), and if the task B does not carry a historical restart task tag, determine that the task B is a pending task that meets the preset conditions.

[0107] Alternatively, still taking task B as an example, the disclosed embodiment can determine the restart times of task B through the restart times label of task B, and determine that task B is a pending task that meets the preset conditions when the restart times of task B is zero.

[0108] The embodiment of the present disclosure can directly determine the pending task with a restart count of zero (i.e., never restarted) as the pending task that meets the preset conditions. This is because the pending task with a restart count of zero has never been restarted, and there will definitely be no corresponding historical restart task for the pending task. Therefore, directly restarting the pending task with a restart count of zero will not cause the problem of repeatedly restarting the pending task.

[0109] Through the above-mentioned method, the embodiment of the present disclosure can reduce the resources required to determine the tasks to be processed that meet the preset conditions while ensuring the accuracy of task processing, avoid unnecessary waste of computing resources and storage resources, ensure that resources are used more efficiently, and thus improve the resource utilization of the overall system.

[0110] Of course, when the number of restarts of the tag data carried by the pending task is not zero, the embodiment of the present disclosure needs to further judge the pending task to determine whether the pending task meets the preset conditions. That is, the embodiment of the present disclosure can also further judge the pending task carrying the historical restart task tag in the following manner.

[0111] Specifically, when the restart count of the pending tasks is not zero, the embodiments of the present disclosure may also use the restart count of the pending tasks and the historical restart task tags to determine the pending tasks that meet the preset conditions from the pending tasks.

[0112] Specifically, when the restart count of the pending task is not zero, the restart count of the pending task and the historical restart task label are used to determine the pending task that meets the preset condition from the pending tasks, including:

[0113] Compare the number of restarts of the pending task with a preset threshold, and when the number of restarts of the pending task is not greater than the preset threshold, use the historical restart task label of the pending task to determine the name data of the historical restart task corresponding to the pending task;

[0114] According to the name data of the historical restart task, search for the historical restart task corresponding to the task to be processed;

[0115] In the case where a historical restart task corresponding to the task to be processed is found, a training result of the historical restart task is determined; and in the case where the training of the historical restart task fails, it is determined that the task to be processed meets a preset condition.

[0116] For example, taking the task to be processed as task B, the preset threshold as m (m is an integer greater than 0), the restart count label as restarts, the historical restart task label as fault-tolerant-jobname, and the training result as status as an example, the embodiment of the present disclosure can use the following steps to determine whether the task to be processed meets the preset conditions:

[0117] Step 2.2.1. Use the restart count label carried by task B, i.e., status=i, to determine that the restart count of task B is i times; then, determine whether the restart count of task B has reached the upper limit, that is, compare the restart count i of task B with the preset threshold m, determine the relationship between the restart count i of task B and the preset threshold m, and when the restart count i is greater than the preset threshold m, determine that task B does not meet the preset conditions, and end this polling; or, when the restart count i is not greater than the preset threshold m, continue to execute step 2.2.2 for task B.

[0118] It should be noted that, when the number of restarts of a pending task is greater than a preset threshold, the subsequent polling of the embodiment of the present disclosure may no longer consider the pending task. The preset threshold may include any non-zero integer, such as 4, 5, 6, etc. Of course, the embodiment of the present disclosure does not limit the specific value of the preset threshold.

[0119] Step 2.2.2, determine the historical restart task label carried by task B, that is, fault-tolerant-jobname, and set fault-tolerant-jobname=B in the historical restart task label i In the case of, determine the name data of the historical restart task carried by the historical restart task tag (i.e., B i ), then, based on the name data of the historical restart task (i.e. B i ), determine that the name of the historical restart task corresponding to task B is B i .

[0120] Step 2.2.3: Search all tasks (i.e. all original tasks and all restarted tasks) to see if there is a task named B. i , and no task named B is found iIn the case of a task named B, it is determined that task B does not meet the preset conditions, and the polling is terminated at this time; or, when a task named B is found i In the case of Task A, continue with step 2.2.4 for that Task B.

[0121] Step 2.2.4: Determine the name as B i The training result of the task named B i The status of the task, and in the task named B i If the status of the task named B is "succeed", it is determined that the task B does not meet the preset condition, and the polling is terminated; or i When the status of task B=failed, it is determined that task B meets the preset condition.

[0122] The disclosed embodiment determines whether the task to be trained meets the preset conditions from multiple angles and in all aspects through the restart times of the task to be processed, the historical restart task labels, and the training status of the historical restart tasks corresponding to the task to be processed. This can prevent the initiation of infinite restart tasks and increase the accuracy of the determined tasks to be processed, thereby improving the stability and reliability of the system.

[0123] Step 3: Use a multi-machine multi-card distributed training method to restart pending tasks that meet the preset conditions, and update the restart count label and historical restart task label of the pending tasks that meet the preset conditions.

[0124] Furthermore, after determining the pending tasks that meet the preset conditions using the above method, the embodiment of the present disclosure can also adopt a distributed training method of multiple machines and multiple cards to restart the pending tasks that meet the preset conditions, and update the restart count label and historical restart task label of the pending tasks that meet the preset conditions.

[0125] Specifically, the distributed training method of multiple machines and multiple cards is adopted to restart the pending tasks that meet the preset conditions, and update the restart count label and historical restart task label of the pending tasks that meet the preset conditions, including:

[0126] Using a multi-machine multi-card distributed training method, restart the pending task that meets the preset conditions to obtain a target restart task corresponding to the pending task that meets the preset conditions; and using the target restart task to update the historical restart task label carried by the pending task that meets the preset conditions; and,

[0127] The restart count corresponding to the pending task that meets the preset condition is increased, and the task restart count label carried by the pending task that meets the preset condition is updated according to the increased restart count.

[0128] For example, the pending task that meets the preset conditions is task B, the restart count label is represented by restarts, the historical restart task label is represented by fault-tolerant-jobname, and the training result is represented by status. In the embodiment of the present disclosure, the following steps can be used to restart the pending task that meets the preset conditions and update the label data carried by the pending task that meets the preset conditions:

[0129] Step 3.1: Based on the training method corresponding to task B, restart task B to obtain a target restart task corresponding to task B, namely task B1.

[0130] Step 3.2, increase the restart count of task B (i.e., increase the restarts corresponding to task B by 1) and update the restart count tag carried by task B to restarts=i+1. At the same time, the disclosed embodiment also needs to use the name of the historical restart task currently corresponding to task B to determine the historical restart task tag of task B, that is, according to the name data of the historical restart task currently corresponding to task B (i.e., B2), and based on the data, update it to the historical restart task tag corresponding to the current task B, i.e., fault-tolerant-jobname=B2.

[0131] The embodiment of the present disclosure updates the label data corresponding to the pending tasks in real time, and tracks and manages each pending task in a concise and clear manner, thereby increasing the stability and reliability of the task processing method proposed in the embodiment of the present disclosure.

[0132] Of course, it should be noted that in order to flexibly adjust the processing strategy for different pending tasks, the embodiment of the present disclosure, before adopting the distributed training method of multiple machines and multiple cards and restarting the pending tasks that meet the preset conditions, also includes:

[0133] Determine the response command corresponding to the pending task that meets the preset conditions;

[0134] The response command is used to determine the training memory and / or training priority corresponding to the task to be processed that meets the preset conditions.

[0135] Among them, for any pending task that meets the preset conditions, the response command can be an arena start command;

[0136] The training priorities include: high priority, medium priority, low priority, etc.

[0137] Specifically, the disclosed embodiment can not only poll and scan multiple original tasks, and select a pending task that meets the preset conditions from the multiple original tasks; it can also read the arena startup command corresponding to the pending task that meets the preset conditions, and use the arena startup command corresponding to the pending task to determine the training memory and / or training priority of the training task that meets the preset conditions; thereafter, based on the training memory and / or training priority corresponding to the pending task that meets the preset conditions, the pending task that meets the preset conditions can be restarted, thereby achieving the purpose of automatically waking up the pending task that meets the preset conditions.

[0138] Through the above method, the embodiment of the present disclosure can automatically wake up the pending tasks that meet the preset conditions, thereby avoiding the problem that the stopped pending tasks cannot be automatically woken up when the machine is allocated, and improving the reliability and training efficiency of the system.

[0139] The disclosed embodiment determines the specific response command corresponding to the pending task that meets the preset conditions, thereby flexibly adjusting the processing strategy according to different task requirements, ensuring that each pending task can receive the processing that best suits its characteristics, thereby improving the flexibility of task processing.

[0140] Of course, it should be noted that the above steps 1 to 3 are only the steps required for one polling. In order to further enhance the accuracy and stability of task processing, the embodiment of the present disclosure can perform regular polling for the above multiple original tasks. That is, after executing steps 1 to 3 for all original tasks, the embodiment of the present disclosure can poll the multiple original tasks again at a preset time interval (such as 10 minutes or 15 minutes, etc.), that is, execute steps 1 to 3 for the multiple original tasks again.

[0141] Furthermore, if Figure 5 As shown, it should be noted that the foundation of the distributed training management system proposed in the embodiment of the present disclosure includes cloud native infrastructure, that is, Figure 5 Cloud native infrastructure shown.

[0142] Specifically, Figure 5 As shown, the foundation of the distributed training management system proposed in the embodiment of the present disclosure (i.e., cloud-native infrastructure) includes the cloud-native solution Kubernetes, the open source component Arena, the open source component Volcano, and the Training-Operator solution.

[0143] Among them, the open source component Arena can simplify the deployment function of distributed tasks; the open source component Volcano can enhance the scheduling function of distributed tasks; the Training-Operator solution can simplify and optimize the process of model training and fine-tuning; and the cloud-native solution Kubernetes can automatically deploy, expand and manage containerization.

[0144] The disclosed embodiments can further optimize the distributed management system by combining with cloud native infrastructure, enhance the scheduling, deployment, expansion and other functions of the distributed management system, and optimize the overall process.

[0145] The present disclosure also proposes a distributed training management system. Figure 6 The structural diagram of the distributed training management system 600 according to an embodiment of the present disclosure includes: an elastic training module 610;

[0146] The elastic training module 610 is used to select tasks to be processed from original tasks; wherein the tasks to be processed include original tasks that have failed in training and can be processed;

[0147] The elastic training module 610 is further used to determine the pending tasks that meet the preset conditions by using the restart count label of the pending tasks and / or the historical restart task label; wherein the preset condition is that the restart count of the pending tasks is zero; or, the restart count of the pending tasks is greater than zero and not greater than a preset threshold, and there is a historical restart task corresponding to the pending tasks that failed training; the restart count indicates the number of times the pending tasks are restarted through the distributed training method of multiple machines and multiple cards;

[0148] The elastic training module 610 is also used to adopt a distributed training method of multiple machines and multiple cards to restart pending tasks that meet preset conditions, and update the restart count label and historical restart task label of the pending tasks that meet the preset conditions.

[0149] In some implementations, the restart count tag is used to indicate the restart count of the pending task;

[0150] The historical restart task tag is used to indicate the name data of the historical restart task corresponding to the task to be processed.

[0151] In some embodiments, the elasticity training module 610 is further used to:

[0152] Determine the processability label and training result of the original task; wherein the processability label is used to indicate the processability of the original task; processability includes processability and unprocessability; and the training result includes training success and training failure;

[0153] By using the processability labels and training results of the original tasks, original tasks that failed in training and are processable are selected from the original tasks as tasks to be processed.

[0154] In some embodiments, the elasticity training module 610 is used to:

[0155] Determine the restart times of the pending task by using the historical restart task tag and / or the restart times tag of the pending task;

[0156] When the number of restarts of the pending task is zero, determining that the pending task meets the preset condition; or,

[0157] When the restart count of the pending task is not zero, the restart count of the pending task and the historical restart task label are used to determine the pending task that meets the preset condition from the pending tasks.

[0158] In some implementations, when the number of restarts of the pending task is not zero, the elastic training module 610 is used to:

[0159] Compare the restart count of the pending task with a preset threshold, and when the restart count of the pending task is not greater than the preset threshold, determine the name data of the historical restart task corresponding to the pending task using the historical restart task label of the pending task;

[0160] According to the name data of the historical restart task, search for the historical restart task corresponding to the task to be processed;

[0161] In the case where a historical restart task corresponding to the task to be processed is found, a training result of the historical restart task is determined; and in the case where the training of the historical restart task fails, it is determined that the task to be processed meets a preset condition.

[0162] In some embodiments, the elasticity training module 610 is used to:

[0163] Using a multi-machine multi-card distributed training method, restart the pending tasks that meet the preset conditions to obtain the target restart tasks corresponding to the pending tasks that meet the preset conditions; and using the target restart tasks, update the historical restart task labels of the pending tasks that meet the preset conditions; and,

[0164] The restart count corresponding to the pending task that meets the preset condition is increased; and according to the increased restart count, the task restart count label carried by the pending task that meets the preset condition is updated.

[0165] In some embodiments, the elasticity training module 610 is further used to:

[0166] Determine the response command corresponding to the pending task that meets the preset conditions;

[0167] The response command is used to determine the training memory and / or training priority corresponding to the task to be processed that meets the preset conditions.

[0168] Figure 7 A structural diagram of a distributed training management system 700 according to an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, in some embodiments, the distributed training management system 700 further includes a graphical user interface 720; wherein,

[0169] The graphical user interface 720 is used to display the display information corresponding to each of the multiple original tasks; wherein,

[0170] The display information includes at least one of the following:

[0171] Training results for the original task;

[0172] The training priority of the original task;

[0173] Training logs for the original task;

[0174] The user who submitted the original task.

[0175] In some implementations, the graphical user interface 720 is also used to edit the training priority of the original task.

[0176] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0177] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0178] Figure 8 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 8 As shown, the electronic device includes: a memory 810 and a processor 820, and the memory 810 stores a computer program that can be run on the processor 820. The number of the memory 810 and the processor 820 can be one or more. The memory 810 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 830, which is used to communicate with external devices and perform data exchange transmission.

[0179] If the memory 810, the processor 820 and the communication interface 830 are implemented independently, the memory 810, the processor 820 and the communication interface 830 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0180] Optionally, in a specific implementation, if the memory 810, the processor 820 and the communication interface 830 are integrated on a chip, the memory 810, the processor 820 and the communication interface 830 can communicate with each other through an internal interface.

[0181] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0182] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0183] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present disclosure are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. Computer-readable storage media may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., digital versatile discs (DVD)), or semiconductor media (e.g., solid state drives (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0184] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0185] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0186] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0187] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0188] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A task processing method applied to multiple machines and multiple cards, comprising: Selecting tasks to be processed from original tasks; wherein the tasks to be processed include original tasks that have failed in training and can be processed; and the original tasks include multi-machine and multi-card tasks; Determine the pending tasks that meet the preset conditions by using the restart times label and / or the historical restart task label of the pending tasks; wherein the preset conditions are that the restart times of the pending tasks are zero; or, the restart times of the pending tasks are greater than zero and not greater than a preset threshold, and there are historical restart tasks with training failures corresponding to the pending tasks; the restart times represent the times of restarting the pending tasks through the distributed training mode of multiple machines and multiple cards; A distributed training method with multiple machines and multiple cards is adopted to restart the pending tasks that meet the preset conditions, and update the historical restart task labels and restart count labels of the pending tasks that meet the preset conditions.

2. The method according to claim 1, wherein: The restart times label is used to indicate the restart times of the pending task; The historical restart task tag is used to indicate the name data of the historical restart task corresponding to the task to be processed.

3. The method according to claim 2, wherein: The step of selecting a task to be processed from the original task includes: Determine the processability label and training result of the original task; wherein the processability label is used to indicate the processability of the original task; the processability includes processable and unprocessable; the training result includes training success and training failure; By using the processability labels and training results of the original tasks, original tasks that failed in training and are processable are selected from the original tasks as tasks to be processed.

4. The method according to claim 3, wherein: The step of using the restart count tag of the pending task and / or the historical restart task tag to determine the pending task that meets the preset condition includes: Determine the restart times of the task to be processed by using the restart times label of the task to be processed and / or the historical restart task labels; When the number of restarts of the pending task is zero, determining that the pending task meets a preset condition; or, When the restart count of the pending task is not zero, the restart count of the pending task and the historical restart task tag are used to determine a pending task that meets a preset condition from the pending tasks.

5. The method according to claim 4, wherein: When the number of restarts of the pending task is not zero, determining the pending task that meets the preset condition from the pending tasks by using the number of restarts of the pending task and the historical restart task label, includes: Compare the number of restarts of the pending task with the preset threshold, and when the number of restarts of the pending task is not greater than the preset threshold, determine the name data of the historical restart task corresponding to the pending task by using the historical restart task label of the pending task; According to the name data of the historical restart task, searching for the historical restart task corresponding to the task to be processed; In the case where a historical restart task corresponding to the task to be processed is found, a training result of the historical restart task is determined; and in the case where the training of the historical restart task fails, it is determined that the task to be processed meets a preset condition.

6. The method according to claim 5, wherein: The distributed training method of multiple machines and multiple cards is adopted to restart the pending tasks that meet the preset conditions, and update the restart count label and the historical restart task label of the pending tasks that meet the preset conditions, including: Adopting a multi-machine multi-card distributed training method, restarting the pending task that meets the preset conditions to obtain a target restart task corresponding to the pending task that meets the preset conditions; and using the target restart task to update the historical restart task label of the pending task that meets the preset conditions; and, The restart count of the pending task that meets the preset condition is increased, and the restart count label of the pending task that meets the preset condition is updated according to the increased restart count.

7. The method according to claim 6, before restarting the pending task that meets the preset conditions in a distributed training mode of multiple machines and multiple cards, further comprises: Determine the response command corresponding to the pending task that meets the preset conditions; The response command is used to determine the training memory and / or training priority corresponding to the task to be processed that meets the preset conditions.

8. A distributed training management system, comprising an elastic training module; wherein: The elastic training module is used to select tasks to be processed from original tasks; wherein the tasks to be processed include original tasks that have failed in training and can be processed; and the original tasks include multi-machine and multi-card tasks; The elastic training module is further used to determine the pending tasks that meet the preset conditions by using the restart times label and / or the historical restart task label of the pending tasks; wherein the preset conditions are that the restart times of the pending tasks are zero; or, the restart times of the pending tasks are greater than zero and not greater than a preset threshold, and there are historical restart tasks corresponding to the pending tasks that have failed training; the restart times represent the number of times the pending tasks are restarted through the distributed training method of multiple machines and multiple cards; The elastic training module is also used to adopt a multi-machine and multi-card training method to restart the pending tasks that meet the preset conditions, and update the historical restart task labels and restart number labels of the pending tasks that meet the preset conditions.

9. The distributed training management system according to claim 8, further comprising a graphical user interface; wherein The graphical user interface is used to display the display information corresponding to each of the multiple original tasks; wherein, The display information includes at least one of the following: The training result of the original task; The training priority of the original task; The training log of the original task; The user who submitted the original task.

10. The distributed training management system according to claim 9, wherein: The graphical user interface is also used to edit the training priority of the original task.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images