Elastic training method and device

Abstract

The invention provides an elastic training method and device. The method comprises the following steps: determining an idle execution module and issuing a training task; if the number of the idle execution modules is larger than or equal to the data parallelism degree of the issued training tasks, target execution modules are determined from the idle execution modules based on the data parallelism degree of the issued training tasks, and the number of the target execution modules is integer multiples of the data parallelism degree corresponding to the issued training tasks; and executing the issued training task on the target execution module. According to the elastic training method and device provided by the invention, idle resources can be utilized to the maximum extent, the execution efficiency of the training task is improved, and the stability and accurate execution of the training task can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a flexibility training method and device. Background Art

[0002] Elastic training can be understood as elastic scaling training, which means dynamically adjusting training resources according to the demand for computing resources during the training process of machine learning and deep learning, so that the computing resources used can be automatically increased or decreased under different load conditions to optimize training efficiency and reduce costs.

[0003] Currently, elastic training is mostly achieved through distributed training frameworks, which are mostly built on Kubernetes (Kubernetes is an open source container orchestration platform, referred to as K8s), and elastically scale up and down when there is an idle Pod (the basic deployment unit in Kubernetes). However, when elastically scaling up and down when there is an idle Pod, the training data needs to be redistributed. If the redistribution is not accurate, it may cause uneven data distribution, which in turn leads to failure of data parallel segmentation, resulting in training failure. Summary of the invention

[0004] The present invention provides an elasticity training method and device to solve the defects existing in the related art.

[0005] The present invention provides an elasticity training method, comprising the following steps: Determine the idle execution module and issue training tasks; If the number of idle execution modules is greater than or equal to the data parallelism of the sent training task, then based on the data parallelism of the sent training task, a target execution module is determined from the idle execution modules, and the number of target execution modules is an integer multiple of the data parallelism corresponding to the sent training task; The sending training task is executed on the target execution module.

[0006] According to a flexible training method provided by the present invention, if the number of the idle execution modules is greater than or equal to the data parallelism of the sent training task, then based on the data parallelism of the sent training task, determining the target execution module from the idle execution modules includes: If the number of idle execution modules is greater than or equal to the minimum number of execution modules for the issued training task, determine whether the number of idle execution modules is greater than or equal to the data parallelism of the issued training task. If so, determine the target execution module from the idle execution modules based on the data parallelism of the issued training task, and the target number of execution modules is less than or equal to the maximum number of execution modules for the issued training task.

[0007] According to a flexible training method provided by the present invention, before executing the sending training task on the target execution module, the method further includes: Listening for the target execution module joining event of issuing the training task; If it is monitored that the number of target execution modules that have been added is an integer multiple of the data parallelism corresponding to the sent training task, the target execution modules that have been added are networked.

[0008] A method for elasticity training provided by the present invention further comprises: If the number of idle execution modules is less than the minimum number of execution modules for issuing training tasks, based on the data parallelism of the ready training tasks, some execution modules of the ready training tasks are deleted, and the number of deleted execution modules is an integer multiple of the data parallelism corresponding to the ready training tasks; If the sum of the number of deleted execution modules and the number of idle execution modules is greater than or equal to the minimum number of execution modules for the issued training task, the deleted execution modules are used as newly added idle execution modules, and the execution is returned. If the number of idle execution modules is greater than or equal to the data parallelism of the issued training task, the target execution module is determined from the idle execution modules based on the data parallelism of the issued training task.

[0009] A method for elasticity training provided by the present invention further comprises: Monitoring the execution module deletion event of the ready training task; If any execution module is deleted, the data parallel group to which the corresponding deleted execution module belongs is determined, and all execution modules in the data parallel group are marked as being in a non-networkable state; After deleting any one of the execution modules, all the execution modules in the data parallel grouping are networked.

[0010] According to a flexible training method provided by the present invention, after deleting the execution modules of the ready training task, the number of remaining execution modules is greater than or equal to the minimum number of execution modules of the ready training task.

[0011] The present invention also provides an elasticity training device, comprising the following modules: A first determination unit, used to determine an idle execution module and issue a training task; A second determining unit is configured to determine a target execution module from the idle execution modules based on the data parallelism of the issued training task if the number of the idle execution modules is greater than or equal to the data parallelism of the issued training task, wherein the number of the target execution modules is an integer multiple of the data parallelism corresponding to the issued training task; A task execution unit is used to execute the sent training task on the target execution module.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the elasticity training methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the elasticity training method as described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the elasticity training method described above is implemented.

[0015] The elastic training method and device provided by the present invention determine the target execution module from the idle execution modules based on the data parallelism of the issued training task, and ensure that the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task, so as to ensure that the target execution modules allocated to each batch corresponding to the issued training task are balanced, avoid the situation where there is insufficient or excessive training resources in any batch, and then ensure the stability of the execution process of the issued training task, and avoid affecting the convergence speed and performance of the model corresponding to the issued training task. In addition, the number of target execution modules in the present invention is an integer multiple of the data parallelism corresponding to the issued training task, so that the target execution modules allocated to each batch are balanced without the need to re-batch division (that is, without changing the data parallelism), and ensure the stability and convergence of the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the elasticity training method provided by the present invention.

[0018] Figure 2 This is the second flow chart of the elasticity training method provided by the present invention.

[0019] Figure 3 It is a flow chart of the execution module networking method provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the elasticity training device provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In order to facilitate understanding of the solution of the present invention, the technical terms involved in the present invention are explained as follows: Pod: In Kubernetes, Pod is the smallest deployable unit. In machine learning and deep learning tasks, Pod can be used to process data in model training in parallel. By creating multiple Pods, training tasks can be distributed to multiple nodes to improve training speed and efficiency.

[0024] At present, in the related technology, elastic expansion and contraction can be performed as long as there is an idle Pod. However, in actual applications, especially when it comes to complex large-scale model training, there are many significant problems.

[0025] First, large-scale models often contain a large number of parameters and complex computing processes, and the computing power of an idle Pod may not be able to meet the training requirements. Through parallel processing of multiple Pods, the computing resources in the cluster can be fully utilized to improve training efficiency. In other words, large-scale model training requires multiple Pods to achieve data parallelism. When there is an idle Pod for scaling, the redistribution of training data is often not accurate enough, which may cause uneven data distribution. This uneven data distribution will cause some Pods to have too much or too little data, which in turn affects the convergence speed of the model and the final training quality, leading to errors and failures in the training process. In addition, some Pods have too little data to fully utilize available resources, resulting in potential resource waste and inefficient training.

[0026] In addition, in a training task, the data set corresponding to the training task is usually divided into multiple batches. Each Pod is usually used to execute a batch of data, that is, the number of Pods is the same as the number of batches, and the number of batches is the data parallelism. Therefore, when there is an idle Pod for scaling, the number of Pods changes, and the data parallelism also changes.

[0027] In addition, during deep learning training, optimizers (such as Adam, SGD, etc.) maintain state information such as momentum and adaptive learning rate. Changing the data parallelism will change the current state information of the model, resulting in inconsistency between the current state information of the model and the state information maintained in the model optimizer, that is, the state information maintained in the optimizer is outdated. If the optimizer updates the model parameters based on outdated state information, it will lead to instability in the training process and even cause the model to fail to converge.

[0028] Figure 1 It is one of the flow charts of the elasticity training method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0029] Step 110: determine an idle execution module and issue a training task.

[0030] Here, the idle execution module can be understood as an unoccupied execution module. The execution module here can also be understood as a training resource for a training task, which is used to execute the training task. The execution module can be the above-mentioned Pod. For example, for a large-scale model training task, the training data can be distributed to multiple Pods to perform parallel operations on multiple Pods. The idle execution module can be obtained through real-time monitoring by the scheduler. Taking the execution module as Pod as an example, the scheduler can be a Kubernetes monitoring tool (such as Prometheus, Grafana), which is used to observe the CPU and memory usage of each Pod, and regards the unused Pod as an idle execution module.

[0031] The dispatching of training tasks refers to the training tasks to be assigned to the execution modules. The dispatching of training tasks can also be understood as training tasks that can be expanded or reduced in capacity. Expansion means that in the process of executing the training tasks, the execution modules can be dynamically added, such as adding Pods to increase the training speed. Reduction means that in the process of executing the training tasks, the execution modules can be dynamically reduced, such as reducing Pods to save resources. Among them, the addition of execution modules here can be to assign the above-mentioned idle execution modules to the training tasks. The training tasks occupy the idle execution modules. At this time, the status of the idle execution modules is occupied, that is, the idle execution modules are converted to non-idle execution modules. In addition, the dispatching of training tasks can also be obtained through real-time monitoring by the above-mentioned scheduler. After monitoring the dispatching of training tasks, the parameters such as data parallelism of the dispatching of training tasks can be obtained through the scheduler.

[0032] Step 120: If the number of idle execution modules is greater than or equal to the data parallelism of the sent training task, the target execution module is determined from the idle execution modules based on the data parallelism of the sent training task, and the number of target execution modules is an integer multiple of the data parallelism corresponding to the sent training task.

[0033] Step 130: Execute and send the training task on the target execution module.

[0034] Specifically, the data parallelism of the training task refers to the number of batches of training data sets corresponding to the training task, and each batch of training data can be executed on a corresponding execution module. After obtaining the training task, the data parallelism of the training task can be determined based on the current training progress, model size, data set characteristics, available hardware resources (such as the number of GPUs and memory), network bandwidth, etc.

[0035] If the number of idle execution modules is greater than or equal to the data parallelism of the issued training task, it indicates that there are idle resources. In order to make full use of the idle resources, the idle execution modules can be allocated to the issued training tasks. For example, some execution modules can be selected from the idle execution modules as target execution modules, and the target execution modules are allocated to the issued training tasks to increase the training resources for the issued training tasks.

[0036] However, if the target execution module is a non-integer multiple of the data parallelism corresponding to the sent training task, there must be at least one target execution module whose data load is different from the data load of the remaining execution modules, that is, the data load is unbalanced. As a result, when processing data, the target execution module will have a delay compared to the other target execution modules, which will lead to training errors and training failures during the execution of the sent training task.

[0037] For example, if the data parallelism of the training task is 4, the training data set corresponding to the training task is divided into 4 batches. If the number of idle execution modules is 6, and 5 idle execution modules are selected as target execution modules, the number of target execution modules is 5, and the number of target execution modules (5) is a non-integer multiple of the data parallelism (4). If the target execution modules are assigned to each batch, there must be a batch with a larger number of target execution modules than the remaining batches, which will cause the processing speed of the training data of this batch to be faster than the processing speed of the training data of the remaining batches, which will lead to data asynchrony during the execution of the training task, and further lead to training errors and training failures during the execution of the training task.

[0038] In this regard, an embodiment of the present invention determines the target execution module from the idle execution modules based on the data parallelism of the issued training task, and ensures that the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task, so that the target execution modules can be evenly distributed for each batch, thereby ensuring that the target execution modules allocated to each batch corresponding to the issued training task are balanced, avoiding insufficient or excessive training resources in any batch, and thus ensuring the stability of the execution process of the issued training task, and avoiding affecting the convergence speed and performance of the model corresponding to the issued training task.

[0039] In addition, in order to ensure that the target execution modules assigned to each batch are balanced, the relevant technology will also change the data parallelism of the training tasks, that is, change the number of batches of the training tasks. Taking the above-mentioned number of target execution modules as 5 as an example, the relevant technology will change the training data set corresponding to the training tasks from 4 batches to 5 batches. Although it ensures that the target execution modules assigned to each batch are balanced, changing the data parallelism will change the current state information of the model, resulting in inconsistency between the current state information of the model and the state information maintained in the model optimizer, that is, the state information maintained in the optimizer is outdated state information. If the optimizer updates the parameters of the model based on outdated state information, it will cause instability in the training process and even cause the model to fail to converge.

[0040] In this regard, in the embodiment of the present invention, the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task, thereby ensuring that the target execution modules allocated to each batch are balanced without the need to redivide the batches (i.e., without changing the data parallelism), thereby ensuring the stability and convergence of the model training process.

[0041] In summary, the elastic training method provided in the embodiment of the present invention determines the target execution module from the idle execution modules based on the data parallelism of the issued training task, and ensures that the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task, thereby ensuring that the target execution modules allocated to each batch corresponding to the issued training task are balanced, avoiding the situation where there is insufficient or excessive training resources in any batch, and then ensuring the stability of the execution process of the issued training task, and avoiding affecting the convergence speed and performance of the model corresponding to the issued training task. In addition, in the embodiment of the present invention, the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task, thereby ensuring that the target execution modules allocated to each batch are balanced without the need to re-batch division (that is, without changing the data parallelism), and ensuring the stability and convergence of the model training process.

[0042] Based on the above embodiment, if the number of idle execution modules is greater than or equal to the data parallelism of the sent training task, then based on the data parallelism of the sent training task, the target execution module is determined from the idle execution modules, including: If the number of idle execution modules is greater than or equal to the minimum number of execution modules for issuing training tasks, then determine whether the number of idle execution modules is greater than or equal to the data parallelism of issuing training tasks. If so, determine the target execution module from the idle execution modules based on the data parallelism of issuing training tasks, and the target number of execution modules is less than or equal to the maximum number of execution modules for issuing training tasks.

[0043] Specifically, the minimum number of execution modules for issuing training tasks refers to the minimum number of resources required to effectively execute the issued training tasks during model training. In other words, the minimum number of execution modules can ensure the normal execution of the issued training tasks.

[0044] If the number of idle execution modules is greater than or equal to the minimum number of execution modules for sending training tasks, it means that there are enough modules to support the normal execution of the sent training tasks. At this time, you can further consider whether the capacity of sending training tasks can be expanded.

[0045] In an embodiment of the present invention, the basis for capacity expansion is to determine whether the number of idle execution modules is greater than or equal to the data parallelism of the issued training tasks. If so, it indicates that the capacity of the issued training tasks can be expanded while ensuring that the execution modules allocated to each batch of the issued training tasks are balanced.

[0046] Under this condition, based on the data parallelism of the issued training tasks, the target execution modules are determined from the idle execution modules. The number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training tasks, thereby ensuring that the execution modules allocated to each batch are balanced when the capacity of the issued training tasks is expanded.

[0047] In addition, considering that there may be multiple training tasks competing for limited resources in the same time period, in order to ensure the smooth execution of other training tasks, the embodiment of the present invention also sets the maximum number of execution modules for issuing training tasks, that is, when the number of execution modules for issuing training tasks is less than or equal to the maximum number of execution modules, the smooth execution of other training tasks can be guaranteed.

[0048] Therefore, when expanding the capacity of the training tasks to be issued, it is also necessary to ensure that the target number of execution modules is less than or equal to the maximum number of execution modules of the training tasks to be issued, so as to ensure the smooth execution of the remaining training tasks.

[0049] Based on any of the above embodiments, the training task is issued on the target execution module, and the process also includes: Listen for the target execution module joining event that issues the training task; If it is monitored that the number of target execution modules that have been added is an integer multiple of the data parallelism corresponding to the sent training task, the target execution modules that have been added are networked.

[0050] Specifically, after determining a plurality of target execution modules, the plurality of target execution modules need to be added to the issued training task, so as to execute the issued training task on the target execution modules.

[0051] By listening to the joining events of the target execution modules, the number of target execution modules that have joined can be monitored in real time. When the number of target execution modules that have joined is an integer multiple of the data parallelism corresponding to the training task being issued, it indicates that all target execution modules have been successfully joined. At this time, the joined target execution modules are networked, so as to avoid the problem that when some target execution modules are not joined or fail to join, other successfully joined target execution modules start working in advance, resulting in asynchronous processing of each batch of data, thus ensuring the atomicity of the training task being issued and improving the correctness and stability of successful training.

[0052] Optionally, the embodiment of the present invention can develop a corresponding plug-in based on the above-mentioned monitoring logic, and integrate the plug-in into a distributed training framework, the distributed training framework is used to use the above-mentioned elastic training method to achieve the expansion of the sent training tasks, and the plug-in is used to ensure the atomicity of the sent training tasks based on the above-mentioned monitoring logic. Among them, the distributed training framework can include PyTorch DDP (Distributed Data Parallel), Horovod, etc.

[0053] Based on any of the above embodiments, the method further includes: If the number of idle execution modules is less than the minimum number of execution modules for issuing training tasks, some execution modules of the ready training tasks are deleted based on the data parallelism of the ready training tasks. The number of deleted execution modules is an integer multiple of the data parallelism corresponding to the ready training tasks. If the sum of the number of deleted execution modules and the number of idle execution modules is greater than or equal to the minimum number of execution modules for issuing training tasks, the deleted execution modules are used as new idle execution modules and returned for execution. If the number of idle execution modules is greater than or equal to the data parallelism of issuing training tasks, the target execution module is determined from the idle execution modules based on the data parallelism of issuing training tasks.

[0054] Specifically, a ready training task refers to a training task that has been assigned an execution module. If the number of idle execution modules is less than the minimum number of execution modules for the training task, it means that the current idle resources are insufficient to support the normal execution of the training task. At this time, it is necessary to consider whether the ready training task can be scaled down to release the idle execution modules to support the normal execution of the training task.

[0055] When releasing idle execution modules from the ready training task, it is also necessary to ensure that the remaining execution modules of each batch of the ready training task are balanced, so as to ensure the stability of the ready training task. In this regard, the embodiment of the present invention deletes some execution modules of the ready training task based on the data parallelism of the ready training task, and the number of deleted execution modules is an integer multiple of the data parallelism corresponding to the ready training task, so as to ensure that the remaining execution modules of each batch of the ready training task are balanced.

[0056] If the sum of the number of deleted execution modules and the number of idle execution modules is greater than or equal to the minimum number of execution modules for issuing training tasks, the deleted execution modules are used as newly added idle execution modules, so as to ensure that the current number of idle execution modules (previous number of idle execution modules + newly added number of idle execution modules) is greater than or equal to the minimum number of execution modules for issuing training tasks, that is, the current number of idle execution modules is sufficient to support the normal execution of the issued training tasks. At this point, the execution can return to step 120 to allocate target execution modules for issuing training tasks.

[0057] Based on any of the above embodiments, the method further includes: Listen for execution module deletion events of ready training tasks; If any execution module is deleted, the corresponding data parallel group to which the deleted execution module belongs is determined, and all execution modules in the data parallel group are marked as being in a non-networkable state; After deleting any execution module, all execution modules in the data parallel grouping are networked.

[0058] Specifically, by monitoring the deletion events of execution modules, it is possible to monitor in real time whether there are execution module deletion events under the ready training tasks. If any execution module deletion is monitored, the corresponding execution module is in a non-networkable state. Under this condition, if the remaining execution modules in the same data parallel group as the execution module are in a networkable state, these remaining execution modules may continue to interact with the deleted execution module according to the original logic, resulting in incomplete data or calculation errors.

[0059] In this regard, when an embodiment of the present invention monitors any execution module, it determines the data parallel group to which the corresponding deleted execution module belongs, and marks all execution modules in the data parallel group as being in a non-networkable state, thereby preventing the execution modules in the data parallel group from continuing to interact with the deleted execution module, resulting in incomplete data or calculation errors.

[0060] After confirming the deletion of the corresponding execution module, the deleted execution module does not exist in the corresponding data parallel group. At this time, all the execution modules in the data parallel group are re-networked. At this time, the networked execution modules will no longer interact with the deleted execution modules, thereby ensuring the atomicity of the ready training tasks and improving the correctness and stability of successful training.

[0061] Optionally, the embodiment of the present invention can develop a corresponding plug-in based on the above-mentioned monitoring logic, and integrate the plug-in into a distributed training framework, the distributed training framework is used to use the above-mentioned elastic training method to achieve the scaling down of the ready training tasks, and the plug-in is used to ensure the atomicity of the ready training tasks based on the above-mentioned monitoring logic. Among them, the distributed training framework can include PyTorch DDP (Distributed Data Parallel), Horovod, etc.

[0062] Based on any of the above embodiments, after deleting the execution modules of the ready training task, the number of remaining execution modules is greater than or equal to the minimum number of execution modules of the ready training task.

[0063] Specifically, the minimum number of execution modules of a ready training task refers to the minimum number of resources required to effectively execute the ready training task during model training. In other words, the minimum number of execution modules can ensure the normal execution of the ready training task.

[0064] After deleting some execution modules of the ready training task, if the number of remaining execution modules is less than the minimum number of execution modules of the ready training task, the normal execution of the ready training task cannot be guaranteed. Therefore, in the embodiment of the present invention, after deleting some execution modules of the ready training task, the number of remaining execution modules is greater than or equal to the minimum number of execution modules of the ready training task, so that the normal execution of the ready training task can be guaranteed.

[0065] Figure 2 FIG. 2 is a flow chart of the elasticity training method provided by the present invention. Figure 2 As shown in the figure, taking the execution module as Pod as an example, determine the number of idle Pods and the triplet [min, max, dp_size] corresponding to the training task being sent. Min represents the minimum number of Pods for sending training tasks, max represents the maximum number of Pods for sending training tasks, and dp_size represents the data parallelism of sending training tasks.

[0066] Determine whether the number of idle Pods is greater than or equal to min. If not, the capacity of the training task cannot be expanded and the allocation fails. If yes, determine whether the number of idle Pods is greater than or equal to dp_size. If yes, allocate dp_size idle Pods from the idle Pods to the training task each time until the number of idle Pods is less than dp_size.

[0067] If the number of free Pods is less than min, delete dp_size' Pods in the ready training task. Before deleting, determine whether the number of remaining Pods of the ready training task is greater than min'. If so, delete it. If not, continue to delete dp_size' Pods in other ready training tasks. Among them, dp_size' represents the data parallelism of the ready training task, and min' represents the minimum number of execution modules of the ready training task. It should be noted that min' Pods must be guaranteed to run under the ready training task, otherwise the job stability will be affected. Therefore, the ready training task can delete at most max'-min' Pods. In addition, there may be multiple ready training tasks. In this case, all ready training tasks have the same priority. Each ready training task first deletes dp_size' Pods, and then determines whether the sum of the number of deleted Pods and the number of free Pods is greater than or equal to min.

[0068] If the sum of the number of deleted Pods and the number of idle Pods is greater than or equal to min, the deleted Pods will be treated as newly added idle Pods, and dp_size idle Pods will be allocated from the idle Pods to the training tasks. If the sum of the number of deleted Pods and the number of idle Pods is less than min, Pods cannot be allocated for the training tasks, and the allocation fails.

[0069] Figure 3 is a flow chart of the execution module networking method provided by the present invention, such as Figure 3 As shown, in the process of executing the above elastic training method, the status of each training task is monitored in real time, including the events of adding and deleting Pods of each training task. In the case of a Pod adding event, when the number of all added Pods is an integer multiple of the data parallelism corresponding to the training task, the status is marked and the corresponding checkpoint is saved so that the processing can be continued at the checkpoint next time, and then all Pods are re-networked. In the case of a Pod deletion event, all Pods in the data parallel group to which the deleted Pod belongs are marked as a non-networkable state. After the Pod is deleted, the status is marked, the corresponding checkpoint is saved, and then all Pods in the data parallel group are re-networked.

[0070] Depend on Figure 2 and Figure 3 It can be seen that the embodiment of the present invention can flexibly scale the training tasks according to the number of idle Pods, make full use of training resources, and avoid resource waste. In addition, the embodiment of the present invention ensures the atomicity of each training task by monitoring the status of each training task in real time, and ensures the stability and correct execution of the training task.

[0071] The elasticity training device provided by the present invention is described below. The elasticity training device described below and the elasticity training method described above can be referred to each other.

[0072] Based on any of the above embodiments, Figure 4 Schematic diagram of the structure of the elasticity training device provided by the present invention. Figure 4 As shown, the device comprises: A first determining unit 410 is used to determine an idle execution module and issue a training task; A second determining unit 420 is configured to determine a target execution module from the idle execution modules based on the data parallelism of the training task if the number of idle execution modules is greater than or equal to the data parallelism of the training task being issued, wherein the number of target execution modules is an integer multiple of the data parallelism corresponding to the training task being issued; The task execution unit 430 is used to execute and send the training task on the target execution module.

[0073] Based on any of the above embodiments, if the number of idle execution modules is greater than or equal to the data parallelism of the sent training task, determining the target execution module from the idle execution modules based on the data parallelism of the sent training task includes: If the number of idle execution modules is greater than or equal to the minimum number of execution modules for issuing training tasks, then determine whether the number of idle execution modules is greater than or equal to the data parallelism of issuing training tasks. If so, determine the target execution module from the idle execution modules based on the data parallelism of issuing training tasks, and the target number of execution modules is less than or equal to the maximum number of execution modules for issuing training tasks.

[0074] Based on any of the above embodiments, the training task is issued on the target execution module, and the process also includes: Listen for the target execution module joining event that issues the training task; If it is monitored that the number of target execution modules that have been added is an integer multiple of the data parallelism corresponding to the sent training task, the target execution modules that have been added are networked.

[0075] Based on any of the above embodiments, it also includes: If the number of idle execution modules is less than the minimum number of execution modules for issuing training tasks, some execution modules of the ready training tasks are deleted based on the data parallelism of the ready training tasks. The number of deleted execution modules is an integer multiple of the data parallelism corresponding to the ready training tasks. If the sum of the number of deleted execution modules and the number of idle execution modules is greater than or equal to the minimum number of execution modules for issuing training tasks, the deleted execution modules are used as new idle execution modules and returned for execution. If the number of idle execution modules is greater than or equal to the data parallelism of issuing training tasks, the target execution module is determined from the idle execution modules based on the data parallelism of issuing training tasks.

[0076] Based on any of the above embodiments, it also includes: Listen for execution module deletion events of ready training tasks; If any execution module is deleted, the corresponding data parallel group to which the deleted execution module belongs is determined, and all execution modules in the data parallel group are marked as being in a non-networkable state; After deleting any execution module, all execution modules in the data parallel grouping are networked.

[0077] Based on any of the above embodiments, after deleting the execution modules of the ready training task, the number of remaining execution modules is greater than or equal to the minimum number of execution modules of the ready training task.

[0078] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the elastic training method, which includes: determining an idle execution module and issuing a training task; if the number of the idle execution modules is greater than or equal to the data parallelism of the issued training task, then based on the data parallelism of the issued training task, determining a target execution module from the idle execution module, the number of the target execution modules being an integer multiple of the data parallelism corresponding to the issued training task; and executing the issued training task on the target execution module.

[0079] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0080] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the elastic training method provided by the above-mentioned methods, which includes: determining an idle execution module and issuing a training task; if the number of idle execution modules is greater than or equal to the data parallelism of the issued training task, then based on the data parallelism of the issued training task, determining a target execution module from the idle execution module, and the number of target execution modules is an integer multiple of the data parallelism corresponding to the issued training task; and executing the issued training task on the target execution module.

[0081] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the elastic training method provided by the above-mentioned methods, the method comprising: determining an idle execution module and issuing a training task; if the number of the idle execution modules is greater than or equal to the data parallelism of the issued training task, then based on the data parallelism of the issued training task, determining a target execution module from the idle execution module, the number of target execution modules being an integer multiple of the data parallelism corresponding to the issued training task; and executing the issued training task on the target execution module.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for elasticity training, characterized in that: include: Determine the idle execution module and issue training tasks; If the number of idle execution modules is greater than or equal to the data parallelism of the sent training task, then based on the data parallelism of the sent training task, a target execution module is determined from the idle execution modules, and the number of target execution modules is an integer multiple of the data parallelism corresponding to the sent training task; The sending training task is executed on the target execution module.

2. The elasticity training method according to claim 1, characterized in that: If the number of the idle execution modules is greater than or equal to the data parallelism of the sent training task, determining a target execution module from the idle execution modules based on the data parallelism of the sent training task includes: If the number of idle execution modules is greater than or equal to the minimum number of execution modules for the issued training task, determine whether the number of idle execution modules is greater than or equal to the data parallelism of the issued training task. If so, determine the target execution module from the idle execution modules based on the data parallelism of the issued training task, and the target number of execution modules is less than or equal to the maximum number of execution modules for the issued training task.

3. The elasticity training method according to claim 1, characterized in that: Before executing the sending training task on the target execution module, the method further includes: Listening for the target execution module joining event of issuing the training task; If it is monitored that the number of target execution modules that have been added is an integer multiple of the data parallelism corresponding to the sent training task, the target execution modules that have been added are networked.

4. The elasticity training method according to claim 1, characterized in that: The method further comprises: If the number of idle execution modules is less than the minimum number of execution modules for issuing training tasks, based on the data parallelism of the ready training tasks, some execution modules of the ready training tasks are deleted, and the number of deleted execution modules is an integer multiple of the data parallelism corresponding to the ready training tasks; If the sum of the number of deleted execution modules and the number of idle execution modules is greater than or equal to the minimum number of execution modules for the issued training task, the deleted execution modules are used as newly added idle execution modules, and the execution is returned. If the number of idle execution modules is greater than or equal to the data parallelism of the issued training task, the target execution module is determined from the idle execution modules based on the data parallelism of the issued training task.

5. The elasticity training method according to claim 4, characterized in that: The method further comprises: Monitoring the execution module deletion event of the ready training task; If any execution module is deleted, the data parallel group to which the corresponding deleted execution module belongs is determined, and all execution modules in the data parallel group are marked as being in a non-networkable state; After deleting any one of the execution modules, all the execution modules in the data parallel grouping are networked.

6. The elasticity training method according to claim 4, characterized in that: After deleting the execution modules of the ready training task, the number of remaining execution modules is greater than or equal to the minimum number of execution modules of the ready training task.

7. An elasticity training device, characterized in that: include: A first determination unit, used to determine an idle execution module and issue a training task; A second determining unit is configured to determine a target execution module from the idle execution modules based on the data parallelism of the issued training task if the number of the idle execution modules is greater than or equal to the data parallelism of the issued training task, wherein the number of the target execution modules is an integer multiple of the data parallelism corresponding to the issued training task; A task execution unit is used to execute the sent training task on the target execution module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the elasticity training method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the elasticity training method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the elasticity training method according to any one of claims 1 to 6 is implemented.

Images