Iterative calculation method and iterative calculation device for model training

Abstract

The embodiment of the invention provides an iterative computation method and an iterative computation device for model training. And during each round of iterative computation, sequentially executing each layer of forward computation task of the to-be-trained model by using the training data of the current iterative process on the GPU equipment, and reversely executing each layer of reverse computation task of the to-be-trained model by using the forward computation result of the last layer. Next, each layer of reverse calculation task is completed, according to a model parameter updating task execution plan, a model parameter updating task of the layer of model structure is distributed to GPU equipment or CPU equipment to execute the model parameter updating task of the layer of model by using gradient information obtained by the layer of reverse calculation task, the model parameter updating task execution plan enables part of the model parameter updating tasks to be executed on the CPU device and the remaining model parameter updating tasks to be executed on the GPU device, and the model parameter updating tasks executed on the GPU device are executed next to the reverse calculation tasks of the model structures of the corresponding layers.

Description

Technical Field

[0001] The embodiments of this specification generally relate to the field of artificial intelligence, and in particular, to an iterative computing method and an iterative computing device for model training. Background Art

[0002] Deep learning models or machine learning models are widely used in application scenarios such as face payment, image recognition, and intelligent customer service. In order to speed up the model training process, GPU devices are usually used to perform most of the model computing tasks in the model training process. With the continuous development of model technology, especially the rise of large models, the number of model parameters has become larger and larger. For example, the number of model parameters of some models has even reached trillions, which has greatly increased the demand for GPU computing resources (such as GPU computing power and GPU video memory) for model training. However, GPU computing resources have grown slowly, making GPU computing resources a resource bottleneck for model training. Summary of the invention

[0003] The embodiments of the present specification provide an iterative computing scheme for model training. Using the iterative computing scheme, the forward computing task of each layer of the model to be trained is sequentially executed on the GPU device, and the forward computing result of the last layer is used to reversely execute the reverse computing task of each layer of the model to be trained, and after each layer of the reverse computing task is completed, the gradient information obtained by the reverse computing task of the layer is allocated to the GPU device or the CPU device to execute the model parameter update task of the layer model according to the task execution status of the GPU device and the CPU device, so that part of the model parameter update task is executed on the CPU device and the remaining model parameter update task is executed on the GPU device, and the model parameter update task executed on the GPU device is executed immediately after the reverse computing task of the corresponding layer model structure, so that at least one rear layer reverse computing task and / or at least one rear layer model parameter update task on the GPU device is synchronously executed during the execution of each model parameter update task on the CPU device, thereby improving the computing resource utilization efficiency in the model training process, and then shortening the iterative computing time of the model training, thereby shortening the model training time.

[0004] According to one aspect of an embodiment of the present specification, an iterative calculation method for model training is provided, comprising: sequentially executing the forward calculation task of each layer of the model to be trained on a GPU device using the training data of the current iteration process and reversely executing the reverse calculation task of each layer of the model to be trained using the forward calculation result of the last layer; and immediately after the reverse calculation task of each layer is completed, according to the model parameter update task execution plan, assigning the model parameter update task of the layer model structure to the GPU device or the CPU device, so that the model parameter update task of the layer model is executed on the GPU device or the CPU device using the gradient information obtained by the reverse calculation task of the layer, wherein the model parameter update task execution plan enables part of the model parameter update tasks to be executed on the CPU device and the remaining model parameter update tasks to be executed on the GPU device, and the model parameter update tasks executed on the GPU device are executed immediately after the reverse calculation tasks of the corresponding layer model structure.

[0005] Optionally, in an example of the above aspect, the model parameter update task execution plan may include a global model parameter update task execution plan for model parameter update tasks of all layer model structures of the model to be trained, and is determined based on task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained, the task running information of each layer of reverse calculation tasks includes task running information when executed on the GPU device, the task running information of each layer of model parameter update tasks includes task running information when executed on the GPU device and task running information when executed on the CPU device, and the task running information includes task execution time.

[0006] Optionally, in an example of the above aspect, the task running information is determined by trial running the model to be trained according to model training configuration information.

[0007] Optionally, in an example of the above aspect, the task running information is determined by providing model training configuration information, model structure information, GPU device computing power capability information and CPU computing power capability information to a task running information generation model.

[0008] Optionally, in an example of the above aspect, the task running information may also include peak memory usage information, and the model parameter update task execution plan may be determined based on the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained and the maximum available memory capacity of the GPU device and the CPU device.

[0009] Optionally, in an example of the above aspect, the model parameter update task execution plan is determined by providing task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained to an execution plan generation model.

[0010] Optionally, in an example of the above aspect, the model parameter update task execution plan may include a local model parameter update task execution plan for the current model parameter update task of the current layer model structure that has just completed the reverse calculation task in the model to be trained, and is determined in real time based on the remaining time for task completion of the CPU device and the task running information when the current model parameter update task is executed on the GPU device.

[0011] Optionally, in an example of the above aspect, when the model parameter update task is executed according to the model parameter update task execution plan, the iterative calculation execution time of the model to be trained is the shortest.

[0012] According to another aspect of an embodiment of the present specification, there is provided an iterative computing device for model training, comprising: at least one processor; a memory coupled to the at least one processor; and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the iterative computing method for model training as described above.

[0013] According to another aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores executable instructions, which, when executed, cause a processor to perform the iterative computing method for model training as described above.

[0014] According to another aspect of an embodiment of the present specification, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the iterative computing method for model training as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A further understanding of the nature and advantages of the content of this specification may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals.

[0016] Figure 1 An example schematic diagram of the iterative computing process of model training based on GPU devices and CPU devices is shown.

[0017] Figure 2 An example flowchart of an iterative computing method for model training according to an embodiment of the present specification is shown.

[0018] Figure 3An example flowchart of a global model parameter update task execution plan generation process according to an embodiment of the present specification is shown.

[0019] Figure 4 An example schematic diagram of the iterative calculation process of model training according to an embodiment of the present specification is shown.

[0020] Figure 5 An example block diagram of an iterative computing device for model training according to an embodiment of the present specification is shown.

[0021] Figure 6 An exemplary schematic diagram of an iterative computing device implemented based on a computer system according to an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0022] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed can be changed without departing from the scope of protection of the contents of this specification. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0023] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0024] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0025] It should be noted that the term "model" is used to refer to various AI models in the field of artificial intelligence, such as deep learning models, machine learning models, etc. Examples of deep learning models include, but are not limited to, deep neural networks (DNN), convolutional neural networks (CNN), residual networks (ResNet), long short-term memory networks (LATM), Transformers, graph neural networks (GNN), large language models (LLM), etc. Examples of machine learning models include, but are not limited to, linear regression models, logistic regression models, decision tree models, support vector machine models, random forests, etc.

[0026] The model training process includes multiple rounds of iterations. In each round of iterations, the training data of this round of iterations is first used to sequentially execute the forward propagation process of each layer of the model structure to be trained. After obtaining the forward propagation result of the last layer of the model structure, the forward propagation result is used to reversely execute the back propagation process of each layer of the model structure to be trained, thereby obtaining the gradient information of each layer of the model structure, and then the gradient information of each layer of the model structure is used to execute the model parameter update process of each layer of the model structure.

[0027] Considering the powerful computing power of GPU devices, in order to speed up the model training process, the iterative calculations of the model training process are usually deployed on GPU devices for execution. As the scale of the model continues to expand, the number of model parameters continues to grow rapidly, so the computing power support required for the model training process is also continuing to grow rapidly. However, GPU computing resources grow slowly, making GPU computing resources a resource bottleneck for model training. Moreover, since the GPU video memory resources are small, they are not enough to store all state quantities (for example, intermediate results and final results) during the model training process, which can easily cause memory overflow (Out of Memory, OOM).

[0028] In order to solve the above problems, a model training scheme based on GPU devices and CPU devices is proposed. In this model training scheme, the forward propagation process and the back propagation process in the iteration process are performed on the GPU device, and then the model parameter update process (also called the model optimization process) in the iteration process is performed on the CPU device, so that part of the state quantity in the model training process can be stored on the CPU device, thereby avoiding memory overflow.

[0029] Figure 1 An example schematic diagram of the iterative calculation process of model training based on GPU devices and CPU devices is shown. Figure 1In the example shown, the model to be trained includes a three-layer model structure. In an iterative calculation process, the forward propagation process of each layer of the model structure is sequentially executed on the GPU device, that is, the forward calculation task FWD1 of the first layer model structure, the forward calculation task FWD2 of the second layer model structure, and the forward calculation task FWD3 of the third layer model structure are sequentially executed. After obtaining the forward calculation result of the forward calculation task FWD3 of the third layer model structure, the reverse propagation process of each layer of the model structure is reversely executed on the GPU device, that is, the reverse calculation task BWD3 of the third layer model structure, the reverse calculation task BWD2 of the second layer model structure, and the reverse calculation task BWD1 of the first layer model structure are sequentially executed. Then, the model parameter update process of the model to be trained is reversely executed on the CPU device, that is, the model parameter update task Step3 of the third layer model structure, the model parameter update task Step2 of the second layer model structure, and the model parameter update task Step1 of the first layer model structure are sequentially executed.

[0030] However, in the iterative process of the above model training, the GPU calculation on the GPU device and the CPU calculation on the CPU device are executed serially, so that when the CPU device executes the model parameter update task, the GPU device is idle, resulting in a waste of computing resources of the GPU device. Depending on the CPU computing power, the time required to execute the model parameter update process on the CPU device is usually 30% to 50% of the time required for the entire iterative process, which will cause an equal proportion of GPU computing power waste, thereby greatly reducing the computing power utilization efficiency and model training efficiency during the model training process.

[0031] To this end, an iterative calculation scheme for model training according to an embodiment of the present specification is proposed. In the iterative calculation scheme, the forward calculation task of each layer of the model to be trained is sequentially executed on the GPU device, and the reverse calculation task of each layer of the model to be trained is executed in reverse order using the forward calculation result of the last layer, and immediately after each layer of reverse calculation task is completed, according to the task execution status of the GPU device and the CPU device, the gradient information obtained by the reverse calculation task of this layer is assigned to the GPU device or the CPU device to execute the model parameter update task of this layer model, so that part of the model parameter update task is executed on the CPU device and the remaining model parameter update task is executed on the GPU device, and the model parameter update task executed on the GPU device is immediately executed after the reverse calculation task of the corresponding layer model structure, thereby during the execution of each model parameter update task on the CPU device, at least one rear-layer reverse calculation task and / or at least one rear-layer model parameter update task on the GPU device can be synchronously executed, thereby improving the efficiency of computing resource utilization during the model training process and shortening the iterative calculation time of model training.

[0032] Figure 2An example flow chart of an iterative computing method 200 for model training according to an embodiment of the present specification is shown. Figure 2 What is shown in FIG. 1 is an iterative process of the model training process executed by the GPU device and the CPU device.

[0033] like Figure 2 As shown, at 210, the forward calculation process of each layer of the model structure to be trained (hereinafter referred to as the forward calculation task of each layer) is sequentially performed on the GPU device using the training data of the current iteration process, thereby obtaining the forward calculation result of the last layer of the model structure. For example, a GPU thread can be created on the GPU device, and the forward calculation task of each layer of the model to be trained is sequentially performed on the GPU thread using the training data of the current iteration process. In some examples, the GPU device can be a GPU device with multiple parallel GPU computing units (for example, GPU cards that can operate independently), or can be multiple independent GPU devices that can calculate in parallel.

[0034] At 220, the forward calculation result of the last layer of the model structure is used on the GPU device to reversely perform the reverse calculation process of each layer of the model structure to be trained (hereinafter referred to as each layer reverse calculation task). After completing the forward calculation task of the last layer of the model structure, the forward calculation result of the last layer of the model structure can be used on the GPU thread to reversely perform each layer of the reverse calculation task.

[0035] At 230, when the reverse calculation tasks of each layer are executed in reverse order on the GPU device, it is determined whether the reverse calculation task of the current layer model structure is completed, that is, whether the reverse calculation result of the current layer model structure (that is, the gradient information of the model parameters of the current layer model structure) is obtained. If the reverse calculation task of the current layer model structure is not completed, return to 220 to continue execution.

[0036] If the reverse calculation task of the current layer model structure is completed, then immediately after the reverse calculation task is completed, at 240, on the GPU device, it is determined whether the model parameter update task of the layer model structure is executed on the GPU device according to the model parameter update task execution plan. The model parameter update task execution plan is used to indicate whether the model parameter update task of each layer model structure in one or more layers of model structures is executed by the GPU device or by the CPU device. The model parameter update task execution plan can be configured so that at least one rear-layer reverse calculation task and / or at least one rear-layer model parameter update task on the GPU device is synchronously executed during the execution of each layer model parameter update task on the CPU device, and at least one model parameter update task is executed on the GPU device.

[0037] In some embodiments, the model parameter update task execution plan may include a global model parameter update task execution plan for model parameter update tasks of all layer model structures of the model to be trained. In some examples, the global model parameter update execution plan may be predetermined.

[0038] Figure 3 An example flow chart of a global model parameter update task execution plan generation process 300 according to an embodiment of the present specification is shown.

[0039] like Figure 3 As shown, at 310, the task running information of each layer of the reverse calculation task and each layer of the model parameter update task of the model to be trained is determined. The task running information of each layer of the reverse calculation task may include the task running information when executed on the GPU device, and the task running information of each layer of the model parameter update task includes the task running information when executed on the GPU device and the task running information when executed on the CPU device. The task running information includes at least the task execution time. In some embodiments, the task running information may also include the peak usage information of the video memory, that is, the maximum video memory size used when executing the task.

[0040] In some embodiments, the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks is determined by trial running the model to be trained according to the model training configuration information. The model training configuration information may include configuration information during model training, such as the batch size (Batch_Size) of training data, etc. For example, two rounds of iterations of the model to be trained can be trial run using training data of a specified batch size. During one round of iteration, all iterative tasks are assigned to the GPU device for execution, and during another round of iteration, the forward calculation tasks of each layer and the reverse calculation tasks of each layer are assigned to the GPU device for execution, and the model parameter update tasks of each layer are assigned to the CPU device for execution, thereby obtaining the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks.

[0041] In some embodiments, a task running information generation model can be pre-trained, and the model training configuration information of the model to be trained, the model structure information of the model to be trained, the GPU device computing power capability information and the CPU computing power capability information are provided to the task running information generation model to determine the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks.

[0042] At 320, a model parameter update task execution plan is determined based on the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained. The determined model parameter update task execution plan can be configured so that part of the model parameter update tasks are executed on the GPU device and the remaining model parameter update tasks are executed on the CPU device, and the model parameter update tasks executed on the GPU device are executed immediately after the reverse calculation tasks of the corresponding layer model structure. According to the model parameter update task execution plan, while the CPU device executes the model parameter update task, the GPU device can synchronously execute at least one rear-layer reverse calculation task and / or at least one rear-layer model parameter update task, thereby improving the computing resource utilization efficiency of the model training process and shortening the iterative calculation time of the model training.

[0043] Since the computing power of the GPU device is much stronger than that of the CPU device, especially when the GPU device is a GPU device with multiple parallel GPU computing units or multiple parallel GPU devices, the task execution time of the reverse computing task on the GPU device is much shorter than the task execution time of the model parameter update task on the CPU device. In this case, when the GPU device completes all layers of reverse computing tasks, there will still be one or more model parameter update tasks to be executed on the CPU device, so that the GPU device is in an idle state when the CPU device continues to execute the model parameter update tasks to be executed. According to the above-mentioned model parameter update task execution plan, part of the model parameter update tasks can be assigned to the GPU device for execution, thereby reducing the workload on the CPU device, thereby shortening the time required for the iterative calculation process, and reducing the idle time of the GPU device, thereby improving the computing resource utilization efficiency of the model training process.

[0044] In some embodiments, the model parameter update task execution plan may be configured such that when the model parameter update task is executed according to the model parameter update task execution plan, the iterative calculation execution time of the model to be trained is the shortest.

[0045] In some embodiments, when the task running information includes the peak usage information of the video memory, the model parameter update task execution plan can be determined based on the task running information of each layer of the reverse calculation task and each layer of the model parameter update task of the to-be-trained model and the maximum available memory capacity of the GPU device and the CPU device. Determining the model parameter update task execution plan in this way can avoid memory overflow.

[0046] In some embodiments, an execution plan generation model may be pre-trained. The learning task of the execution plan generation model during model training may, for example, include the shortest iterative calculation execution time of the model to be trained. After the execution plan generation model is trained, the task running information of each layer of the reverse calculation task and each layer of the model parameter update task of the model to be trained may be provided to the execution plan generation model to determine the model parameter update task execution plan.

[0047] In some embodiments, the model parameter update task execution plan can be predetermined and stored in the GPU device or the task scheduling unit for use in iterative calculations. When the model parameter update task execution plan is stored in the task scheduling unit, after the GPU device completes each layer of reverse calculation tasks, it sends a reverse calculation task completion message to the scheduling unit, and then the scheduling unit sends a task scheduling message to the GPU device to instruct the GPU device or the CPU device to execute the corresponding model parameter update task according to the model parameter update task execution plan. In response to receiving the task scheduling message, the model parameter update task of the model structure of this layer is executed on the GPU device using the gradient information obtained by the reverse calculation task of this layer, or the gradient information obtained by the reverse calculation task of this layer is sent to the CPU device, so that the CPU device uses the gradient information obtained by the reverse calculation task of this layer to execute the model parameter update task of this layer of model structure.

[0048] In some embodiments, the model parameter update task execution plan may include a local model parameter update task execution plan for the current model parameter update task of the current layer model structure that has just completed the reverse calculation task in the model to be trained. The local model parameter update task execution plan can be determined in real time based on the remaining time for task completion of the CPU device (i.e., the remaining time for task completion of the model parameter update task being executed on the CPU device) and the task running information when the current model parameter update task is executed on the GPU device. According to this model parameter update task execution plan, the current model parameter update task can be assigned to the computing device (GPU device or CPU device) that completes the task calculation faster for execution, thereby improving the iterative computing efficiency.

[0049] Back to Figure 2 When it is determined to execute on a GPU device according to the model parameter update task execution plan, the model parameter update task of the model structure of the layer is executed on the GPU device using the gradient information of the model parameters of the model structure of the layer at 250. For example, the model parameter update task of the model structure of the layer is executed on a GPU thread using the gradient information of the model parameters of the model structure of the layer.

[0050] When the execution is determined on the CPU device according to the model parameter update task execution plan, at 260, the gradient information of the model parameters of the layer model structure is sent to the CPU device, and at 270, the model parameter update task of the layer model structure is executed on the CPU device using the gradient information of the model parameters of the layer model structure. For example, a CPU thread is created on the CPU device, and the model parameter update task of the layer model structure is executed on the created CPU thread using the gradient information of the model parameters of the layer model structure. In some embodiments, when the CPU device (CPU thread) is idle, the model parameter update task of the layer model structure can be executed immediately. In some embodiments, when the CPU device (CPU thread) is executing the model parameter update task of the previous layer model structure, the model parameter update task of the layer model structure can be put into the task execution queue of the CPU device (CPU thread) and wait. Once the model parameter update task of the previous layer model structure is completed, the model parameter update task of the layer model structure is executed on the CPU device (CPU thread).

[0051] At 280, it is determined whether the model parameter update task allocation is completed for all layer model structures. If the model parameter update task allocation is not completed for all layer model structures, the process returns to 220 to continue to complete the subsequent reverse calculation task, thereby performing the next model parameter update task allocation and execution.

[0052] Figure 4 An example schematic diagram of the iterative calculation process of model training according to an embodiment of the present specification is shown. Figure 4 In the example, the model to be trained includes a three-layer model structure.

[0053] like Figure 4 As shown, in one round of iterative calculation, the forward calculation task FWD1 of the first-layer model structure, the forward calculation task FWD2 of the second-layer model structure, and the forward calculation task FWD3 of the third-layer model structure are sequentially executed on the GPU device. After obtaining the forward calculation result of the forward calculation task FWD3 of the third-layer model structure, the reverse calculation task BWD3 of the third-layer model structure, the reverse calculation task BWD2 of the second-layer model structure, the reverse calculation task BWD1 of the first-layer model structure, and the model parameter update task Step1 of the first-layer model structure are sequentially executed on the GPU device. While the reverse calculation task BWD2 of the second-layer model structure, the reverse calculation task BWD1 of the first-layer model structure, and the model parameter update task Step1 of the first-layer model structure are sequentially executed on the GPU device, the model parameter update task Step3 of the third-layer model structure and the model parameter update task Step2 of the second-layer model structure are simultaneously sequentially executed on the CPU device.

[0054] As above Figures 2 to 4 An iterative calculation process for model training according to an embodiment of the present specification is described. Using the above iterative calculation process, part of the model parameter update tasks are executed on the CPU device and the remaining model parameter update tasks are executed on the GPU device, and the model parameter update tasks executed on the GPU device are executed immediately after the reverse calculation tasks of the corresponding layer model structure, so that during the execution of each model parameter update task on the CPU device, at least one rear-layer reverse calculation task and / or at least one rear-layer model parameter update task on the GPU device is synchronously executed, thereby improving the efficiency of computing resource utilization during the model training process, thereby shortening the iterative calculation time of the model training, and thus shortening the model training time.

[0055] By utilizing the above-mentioned iterative calculation process, a global model parameter update task execution plan for the model parameter update tasks of all layers of the model structure of the model to be trained is determined according to the task running information of the reverse calculation tasks of each layer and the model parameter update tasks of each layer of the model to be trained. This can achieve global optimization of the iterative calculation process, thereby further improving the computing resource utilization efficiency of the model training process and shortening the iterative calculation time of model training.

[0056] By utilizing the above iterative calculation process and taking the peak value of the video memory usage of the computing task into consideration when determining the execution plan for updating the model parameters, it can be ensured that no memory overflow occurs when executing the iterative calculation of the model.

[0057] Figure 5 FIG. 5 shows an example block diagram of an iterative computing device 500 for model training according to an embodiment of the present specification. Figure 5 As shown, the iterative computing device 500 includes a first task execution unit 510 located on a GPU device, a second task execution unit 520 located on a CPU device, and a task scheduling unit 530.

[0058] In each round of iterative calculation, the first task execution unit 510 uses the training data of the current iteration process to sequentially execute the forward calculation task of each layer of the model to be trained and uses the forward calculation result of the last layer to reversely execute the reverse calculation task of each layer of the model to be trained.

[0059] After each layer of reverse calculation tasks is completed, the scheduling unit 530 assigns the model parameter update task of the layer of model structure to the first task execution unit 510 on the GPU device or to the second task execution unit 520 on the CPU device according to the model parameter update task execution plan.

[0060] In response to the model parameter updating task being assigned to the first task execution unit 510, the first task execution unit 510 performs the model parameter updating task of the model structure layer using the gradient information obtained from the reverse calculation task of the layer.

[0061] In response to the model parameter update task being assigned to the second task execution unit 520, if the CPU device is idle, the second task execution unit 520 immediately uses the gradient information obtained by the reverse calculation task of this layer to execute the model parameter update task of the model structure of this layer. If the CPU device is executing the model parameter update task of the previous layer model structure, the model parameter update task of the model structure of this layer is placed in the task execution queue of the CPU device and waits. Once the model parameter update task of the previous layer model structure is completed, the second task execution unit 520 executes the model parameter update task of the model structure of this layer using the gradient information obtained by the reverse calculation task of this layer.

[0062] In some embodiments, the iterative computing device 500 may further include a task execution plan generation unit (not shown). In some embodiments, the task execution plan generation unit is configured to determine a model parameter update task execution plan (global model parameter update task execution plan) based on task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained. The determined model parameter update task execution plan may be configured to cause part of the model parameter update tasks to be executed on the GPU device and the remaining model parameter update tasks to be executed on the CPU device, and synchronously execute at least one rear-layer reverse calculation task and / or at least one rear-layer model parameter update task on the GPU device during the execution of each layer of model parameter update tasks on the CPU device.

[0063] In some embodiments, the task execution plan generation unit can be configured to, in response to completing the reverse calculation task of a layer of model structure, determine in real time the model parameter update task execution plan (local model parameter update task execution plan) for the model parameter update task of this layer of model structure based on the remaining task completion time of the CPU device and the task running information when the model parameter update task of this layer of model structure is executed on the GPU device.

[0064] As above Figures 1 to 5 , describes an iterative calculation method and an iterative calculation device for model training according to an embodiment of this specification. The above-mentioned iterative calculation device can be implemented by hardware, or by software or a combination of hardware and software.

[0065] Figure 6 FIG. 6 shows an example schematic diagram of an iterative computing device 600 implemented based on a computer system according to an embodiment of the present specification. Figure 6As shown, the iterative computing device 600 may include at least one processor 610, a memory (e.g., a non-volatile memory) 620, a memory 630, and a communication interface 640, and the at least one processor 610, the memory 620, the memory 630, and the communication interface 640 are connected together via a bus 660. The at least one processor 610 executes at least one computer-readable instruction (i.e., the above-mentioned element implemented in the form of software) stored or encoded in the memory.

[0066] In one embodiment, computer executable instructions are stored in a memory, which, when executed, cause at least one processor 610 to: sequentially execute the forward calculation task of each layer of the model to be trained on a GPU device using the training data of the current iteration process and reversely execute the reverse calculation task of each layer of the model to be trained using the forward calculation result of the last layer; and immediately after the reverse calculation task of each layer is completed, according to the model parameter update task execution plan, assign the model parameter update task of the layer model structure to the GPU device or the CPU device to execute the model parameter update task of the layer model using the gradient information obtained by the reverse calculation task of the layer, wherein the model parameter update task execution plan causes part of the model parameter update tasks to be executed on the CPU device and the remaining model parameter update tasks to be executed on the GPU device, and the model parameter update tasks executed on the GPU device are executed immediately after the reverse calculation tasks of the corresponding layer model structure.

[0067] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 610 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 5 Describes the various operations and functions.

[0068] According to one embodiment, a program product such as a machine-readable medium (e.g., a non-transitory machine-readable medium) is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of this specification. Figure 1-Figure 5 Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.

[0069] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0070] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.

[0071] According to one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the processor performs the above combination of the various embodiments of this specification. Figure 1-Figure 5 Describes the various operations and functions.

[0072] Those skilled in the art should understand that the various embodiments disclosed above can be modified and altered in various ways without departing from the essence of the invention. Therefore, the protection scope of the present invention should be defined by the appended claims.

[0073] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0074] In the above embodiments, the hardware unit or module can be implemented mechanically or electrically. For example, a hardware unit, module or processor can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit or processor can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0075] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0076] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. An iterative calculation method for model training, comprising: On the GPU device, use the training data of the current iteration process to sequentially perform the forward calculation task of each layer of the model to be trained and use the forward calculation result of the last layer to reversely perform the reverse calculation task of each layer of the model to be trained; as well as After each layer of reverse calculation task is completed, the model parameter update task of the layer of model structure is assigned to the GPU device or CPU device according to the model parameter update task execution plan, so that the model parameter update task of the layer of model is executed on the GPU device or CPU device using the gradient information obtained by the reverse calculation task of the layer. The model parameter update task execution plan enables part of the model parameter update tasks to be executed on the CPU device and the remaining model parameter update tasks to be executed on the GPU device, and the model parameter update tasks executed on the GPU device are immediately followed by the reverse calculation tasks of the corresponding layer model structure.

2. The iterative calculation method according to claim 1, wherein: The model parameter update task execution plan includes a global model parameter update task execution plan for model parameter update tasks of all layer model structures of the model to be trained, and is determined based on task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained, wherein the task running information of each layer of reverse calculation tasks includes task running information when executed on the GPU device, the task running information of each layer of model parameter update tasks includes task running information when executed on the GPU device and task running information when executed on the CPU device, and the task running information includes task execution time.

3. The iterative calculation method according to claim 2, wherein: The task running information is determined by trial running the model to be trained according to the model training configuration information.

4. The iterative calculation method according to claim 2, wherein: The task running information is determined by providing model training configuration information, model structure information, GPU device computing power capability information and CPU computing power capability information to the task running information generation model.

5. The iterative calculation method according to claim 2, wherein: The task running information also includes peak memory usage information, and the model parameter update task execution plan is determined based on the task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained and the maximum available memory capacity of the GPU device and the CPU device.

6. The iterative calculation method according to claim 2, wherein: The model parameter update task execution plan is determined by providing task running information of each layer of reverse calculation tasks and each layer of model parameter update tasks of the model to be trained to an execution plan generation model.

7. The iterative calculation method according to claim 1, wherein: When the model parameter update task is executed according to the model parameter update task execution plan, the iterative calculation execution time of the model to be trained is the shortest.

8. The iterative calculation method according to claim 1, wherein: The model parameter update task execution plan includes a local model parameter update task execution plan for the current model parameter update task of the current layer model structure that has just completed the reverse calculation task in the model to be trained, and is determined in real time based on the remaining time for task completion of the CPU device and the task running information when the current model parameter update task is executed on the GPU device.

9. An iterative computing device for model training, comprising: at least one processor; a memory coupled to the at least one processor; as well as A computer program stored in the memory, wherein the at least one processor executes the computer program to implement the iterative computing method for model training as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing executable instructions, which, when executed, cause a processor to perform the iterative computing method for model training as described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the iterative computing method for model training as described in any one of claims 1 to 8.

Images