Model training method and device, equipment and medium
By dividing modules in the backpropagation of the model and synchronously performing communication processing and gradient accumulation, the additional overhead problem introduced by the video memory optimization method in distributed processing is solved, and more efficient model training performance and video memory management are achieved.
Patent Information
- Application Number
- CN202411726193.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-11
AI Technical Summary
In distributed processing, common video memory optimization methods will introduce additional communication overhead and copy overhead, resulting in poor performance in model training.
By performing communication processing in the backpropagation of the model, the model is divided into multiple modules, and after each module completes backpropagation, communication processing and reverse gradient accumulation are performed simultaneously, thereby masking the time-consuming of gradient accumulation in the communication processing.
This method shortens the time-consuming model training, improves the efficiency of model training, improves the model training performance, and effectively reduces the peak model memory by timely deleting gradient temporary video memory.
Smart Images
Figure CN119599086A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, device, equipment and medium. Background Art
[0002] The rapid development of artificial intelligence technology, especially the popularity of large language models (LLM), has led to a large increase in model parameters, which requires high computing power support. Therefore, models are often deployed on multiple GPUs (Graphics Processing Units) for distributed processing to improve speed and real-time performance.
[0003] In distributed processing, 3D parallelism and memory offload are often used to optimize video memory. However, these memory optimization methods will introduce additional communication overhead and copy overhead, resulting in poor performance of model training. Summary of the invention
[0004] The present invention provides a model training method, apparatus, device and medium to solve the defect in the related art that additional overhead is introduced during distributed processing, resulting in poor training performance.
[0005] The present invention provides a model training method, comprising: Dividing the model into a plurality of modules based on the communication processing performed by the model during back propagation; Determine a target module from the multiple modules, and perform back propagation on the target module; When a back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result; After the communication processing and the reverse gradient accumulation are completed, returning to determine a new target module from the multiple modules until the multiple modules complete reverse propagation; Based on the result of the reverse gradient accumulation, parameter iteration is performed on the model.
[0006] According to a model training method provided by the present invention, the communication processing performed by the model in back propagation is based on dividing the model into multiple modules, including: The model is divided into a plurality of modules by taking the starting position of the model for communication processing in the back propagation as the segmentation position of the modules in the model.
[0007] A model training method provided by the present invention also includes: When it is determined that the reverse gradient accumulation is completed, the gradient in the back-propagation result is deleted from the memory of the artificial intelligence chip, and the artificial intelligence chip is used for the model training.
[0008] A model training method provided by the present invention also includes: Performing forward propagation on the model to obtain a forward propagation result of the model; The performing back propagation on the target module includes: In the case that the target module is the first module among the multiple modules to perform back propagation, the forward propagation result is used as the back propagation input to perform back propagation on the target module.
[0009] According to a model training method provided by the present invention, the back propagation of the target module includes: In the case that the target module is not the first module among the multiple modules to perform back propagation, the data obtained from the previous communication process is used as the back propagation input to perform back propagation on the target module.
[0010] According to a model training method provided by the present invention, determining a new target module from the multiple modules includes: Based on the execution order of back propagation of each module in the model, a next module executed after the target module is determined from the multiple modules as a new target module.
[0011] The present invention also provides a model training device, comprising: A partitioning unit, for partitioning the model into a plurality of modules based on communication processing performed by the model in back propagation; A back propagation unit, used to determine a target module from the multiple modules and perform back propagation on the target module; A concurrent unit, configured to simultaneously perform communication processing and reverse gradient accumulation based on the back-propagation result when a back-propagation result of the target module is obtained; A selection unit, configured to return to determine a new target module from the multiple modules after the communication processing and the reverse gradient accumulation are completed, until the multiple modules complete reverse propagation; An iteration unit is used to iterate the parameters of the model based on the results of reverse gradient accumulation of the multiple modules.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model training method described above is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the model training methods described above.
[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the model training methods described above.
[0015] The model training method, apparatus, device and medium provided by the present invention divide the model into multiple modules based on the communication processing performed by the model during back propagation, and synchronously perform communication processing and reverse gradient accumulation in each module after the back propagation is completed, thereby masking the time consumption of reverse gradient accumulation in model training in the communication processing time consumption, thereby shortening the model training time consumption, improving the model training efficiency, and thus improving the model training performance. At the same time, timely deleting the gradient temporary memory can effectively reduce the model peak memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 It is one of the flow charts of the model training method provided by the present invention.
[0018] Figure 2 This is the second flow chart of the model training method provided by the present invention.
[0019] Figure 3 This is the third flow chart of the model training method provided by the present invention.
[0020] Figure 4 It is a structural schematic diagram of the model training device provided by the present invention.
[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] With the rapid development of artificial intelligence technology, especially the rise of large-scale language models, the number of model parameters has increased dramatically to support more complex and accurate natural language processing tasks. The huge number of parameters also means high demand for computing resources. It is usually necessary to deploy the model to multiple GPUs or even multiple servers for distributed reasoning to accelerate the processing process and meet real-time requirements.
[0024] In distributed reasoning, common video memory optimization methods include 3D parallelism and video memory offload.
[0025] Among them, 3D parallelism is a technology that divides models and data into multiple parts and processes them in parallel on multiple GPUs. It mainly includes three methods: Data Parallelism (DP): Each GPU processes a subset of the dataset, and the model is fully replicated on all GPUs. In each iteration, each GPU independently calculates the gradient, and then these gradients are reduced to update the model parameters.
[0026] Tensor Parallelism (TP): Different parts of the model are assigned to different GPUs. Each GPU processes a part of the model and exchanges necessary intermediate results with other GPUs to complete forward and backward propagation.
[0027] Pipeline Parallelism (PP): The model is divided into multiple stages, each of which is executed on a different GPU. Data is passed between stages in a pipeline manner, and each GPU processes different stages of different data batches simultaneously.
[0028] In 3D parallelism, whether it is data parallelism, tensor parallelism or pipeline parallelism, additional communication overhead and copy overhead will be introduced, resulting in limited performance of model training.
[0029] Memory offload is a memory optimization technology that transfers currently unneeded tensors from the GPU to the CPU (Central Processing Unit) memory, and then copies them back from the CPU memory to the GPU when they are needed. It can be understood that transferring currently unneeded tensors from the GPU to the CPU memory can free up GPU memory for other operations. Although memory offload can effectively reduce the use of video memory, it will increase the data transmission overhead between the GPU and the CPU, which will also lead to a decrease in model training performance.
[0030] To solve the problems in the above memory optimization methods, the Megatron framework proposes the shard offload technology: Unlike transferring the entire tensor to the CPU, shard offload splits the tensor into multiple parts during forward propagation, and each GPU is only responsible for offloading its stored part of the tensor to the CPU. During reverse propagation, each GPU only restores its own stored part and reassembles it into a complete tensor through the Gather operation. As a result, shard offload reduces the total amount of data transmission each time and reduces communication overhead.
[0031] However, in the shard offload technology, each tensor splitting and tensor combination process will introduce new performance overhead, which makes the shard offload technology also have an adverse impact on the performance of model training.
[0032] In view of the above problems, the present invention provides a model training method. Figure 1 It is one of the flow charts of the model training method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, based on the communication processing performed by the model in back propagation, the model is divided into multiple modules.
[0033] Here, the model is the model that needs to be trained, which can be a large language model, or other models with a large parameter scale that require distributed processing, such as a Transformer model.
[0034] Backpropagation is an algorithm used in the training process of neural network models to calculate the gradient (i.e., the derivative of the loss function with respect to the network parameters), and then update the network weights through optimization algorithms such as gradient descent.
[0035] In a distributed processing mechanism, communication processing is usually required during the back propagation process for the model. For example, in the back propagation process based on shard offload, after each artificial intelligence chip recovers or obtains the part of the tensor corresponding to itself, it needs to communicate with other artificial intelligence chips, and then reassemble into a complete tensor through the Gather operation. The artificial intelligence chip here can be a GPU, or it can also be a TPU (Tensor Processing Unit), NPU (Neural network Processing Unit), DPU (Deeplearning Processing Unit), APU (Accelerated Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Unit), etc., and the embodiment of the present invention does not specifically limit this.
[0036] Since communication processing may be performed once or multiple times during the back propagation process, and the nodes executed during the back propagation process are directly related to the parts of the model executed during the back propagation process, the model can be divided based on the location where the model performs communication processing during the back propagation, thereby obtaining multiple modules in the model.
[0037] It is understandable that multiple modules can be combined into a model, a module is specifically a part of the model, and the number of modules is directly related to the number of communication processing. For example, if the number of communication processing is n, the number of modules obtained by division can be n+1.
[0038] Step 120, determining a target module from the multiple modules, and performing back propagation on the target module.
[0039] Specifically, after the model is divided into multiple modules, a target module can be determined from the multiple modules. The target module here is the module currently performing back propagation.
[0040] In the case where the target module is the first module to perform back propagation, the module arranged at the end of the model can be directly determined from multiple modules as the target module. The module arranged at the end of the model referred to here is the last module executed when forward propagation is performed on the model, that is, the module closest to the output of the model; in the case where the target module is not the first module to perform back propagation, the module arranged before the last module to perform back propagation can be directly determined from multiple modules as the target module, thereby achieving the purpose of performing back propagation one by one according to the arrangement order of each module in the model; the module arranged before the last module to perform back propagation referred to here is the module that performs forward propagation before the module in the process of forward propagation for the model. For example, there are 5 modules in the model, and modules 1, 2, 3, 4, and 5 are executed one by one during forward propagation. Then, corresponding to the reverse propagation, module 5 is the first module to perform back propagation. After module 5 performs back propagation, module 4 is the next module to perform back propagation, that is, when performing back propagation, the order of modules 5, 4, 3, 2, and 1 is executed.
[0041] Step 130, when the back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result.
[0042] Specifically, after back propagation is performed on the target module, the back propagation result of the target module can be obtained. Here, the back propagation result may include the data output by the target module after back propagation, and may also include the gradient of the target module.
[0043] After obtaining the back-propagation results of the target module, communication processing can be performed on the back-propagation results (specifically, it can be Reduce_scatter), so that each artificial intelligence chip involved in the back-propagation process can communicate the results obtained by back-propagating to the target module, so as to realize the tensor combination required for the back-propagation of the next target module.
[0044] While performing communication processing on the back-propagation results, gradient accumulation can be performed based on the back-propagation results. It can be understood that the gradient accumulation referred to here is to perform gradient accumulation on the back-propagation results calculated on the artificial intelligence chip, and this process does not involve communication processing between different artificial intelligence chips, that is, the gradient accumulation here can be understood as partial reverse gradient accumulation.
[0045] It can be understood that, for multiple modules divided by communication processing in back propagation, after one module performs back propagation, communication processing for back propagation is a necessary operation for back propagation of the next module. That is, after the back propagation of the target module is completed, the time consumption caused by the communication processing for the back propagation result is inevitable, and at the same time, the reverse gradient accumulation that is not strongly dependent on the communication processing can achieve the concurrency of communication processing and reverse gradient accumulation, so that the communication time consumption can cover the reverse gradient accumulation time consumption, that is, in the back propagation process of the model, it is no longer necessary to consume extra time for reverse gradient accumulation.
[0046] Step 140, after the communication processing and the reverse gradient accumulation are completed, return to determine a new target module from the multiple modules until the multiple modules have completed reverse propagation.
[0047] Specifically, after the communication processing and reverse gradient accumulation of the back-propagation results of the target module have been completed, the process returns to step 120, and a new target module is selected from multiple modules for back-propagation until all modules in the model have completed back-propagation and the concurrent communication processing and reverse gradient accumulation after back-propagation have been completed.
[0048] Step 150: performing parameter iteration on the model based on the result of the reverse gradient accumulation.
[0049] Specifically, after all modules have completed the reverse gradient accumulation of the back-propagation results, the reverse gradient accumulation results at each artificial intelligence chip can be obtained. Subsequently, based on the reverse gradient accumulation results at each artificial intelligence chip, the gradient accumulation results obtained by back-propagating the model can be determined, and the parameters of the entire model can be iterated to complete the model training.
[0050] In the method provided by the embodiment of the present invention, based on the communication processing performed by the model during back propagation, the model is divided into multiple modules, and communication processing and reverse gradient accumulation are performed synchronously after the back propagation in each module is completed, thereby masking the time consumption of reverse gradient accumulation in model training in the communication processing time consumption, thereby shortening the model training time consumption, improving the model training efficiency, and thus improving the model training performance. At the same time, the gradient temporary memory is deleted in time, which can also effectively reduce the peak memory consumption.
[0051] Moreover, as the scale of the model increases, the number of artificial intelligence chip clusters required for model training will also increase, which makes the communication time for model training often greater than the computational time for gradient accumulation. The communication time better covers the time consumption of reverse gradient accumulation, and the performance improvement of the model training method provided by the embodiment of the present invention is greater.
[0052] Based on the above embodiment, in step 110, the communication processing performed by the model in the back propagation is performed to divide the model into multiple modules, including: The model is divided into a plurality of modules by taking the starting position of the model for communication processing in the back propagation as the segmentation position of the modules in the model.
[0053] Specifically, when dividing the model, the starting position of the model for communication processing in the back propagation can be determined. Here, the starting position for communication processing is the position where the back propagation runs in the model each time the model starts communication processing during the back propagation process. The module division can be achieved by using the starting position for communication processing as the segmentation position when dividing the module.
[0054] In the multiple modules obtained by the division, communication processing is naturally connected after back propagation in each module, so that the time consumption of this part of communication processing can be used to cover the time consumption of part of the reverse gradient accumulation in model training, thereby achieving the effect of shortening the model training time and improving the model training efficiency.
[0055] Specifically, when dividing the model, the starting position of the communication processing in the back propagation of the model can be captured through the Hook mechanism, and the model can be divided based on the captured position. The Hook mechanism here, namely PyTorch Hook, is a mechanism in PyTorch, which is used to obtain information of the intermediate layer, modify the behavior of the model, etc. during the forward propagation and backward propagation of the neural network model. In an embodiment of the present invention, the module before the communication processing in the back propagation can be captured through the Hook mechanism, thereby realizing model division.
[0056] Based on any of the above embodiments, after step 130, the method further includes: When it is determined that the reverse gradient accumulation is completed, the gradient in the back-propagation result is deleted from the memory of the artificial intelligence chip, and the artificial intelligence chip is used for the model training.
[0057] Specifically, the above-mentioned model training process, including back propagation, communication processing based on the back propagation results after back propagation, and reverse gradient accumulation, can all be completed by artificial intelligence chips.
[0058] After back propagation for the target module, the AI chip can obtain the back propagation result for the target module and store the back propagation result for the target module into the memory. Taking the AI chip as a GPU as an example, the memory of the GPU is the temporary storage on the GPU for data storage, which is usually called video memory.
[0059] After obtaining the back-propagation results, the artificial intelligence chip can concurrently perform communication processing and reverse gradient accumulation based on the back-propagation results. Here, the process of reverse gradient accumulation is the process of accumulating low-precision gradients to high-precision gradients. When the reverse gradient accumulation is completed, the artificial intelligence chip can immediately delete the gradients in the back-propagation results stored in the memory, thereby releasing the temporary video memory occupied by the low-precision gradients. By timely clearing temporary tensors such as gradients generated during model training, the video memory peak value during model training can be effectively reduced, thereby achieving video memory optimization for model training under distributed processing.
[0060] Based on any of the above embodiments, the method further includes: Performing forward propagation on the model to obtain a forward propagation result of the model; In step 120, the back propagation of the target module includes: In the case that the target module is the first module among the multiple modules to perform back propagation, the forward propagation result is used as the back propagation input to perform back propagation on the target module.
[0061] Specifically, in the process of model training, before backpropagation, it is necessary to perform forward propagation on the model. Forward propagation here is a calculation process of the neural network model, which is used to calculate the output value based on the input data and model parameters. The forward propagation result obtained is the output value of the model obtained by forward propagation on the model.
[0062] It can be understood that in the process of back propagation for the model, specifically in the process of back propagating each module in the model in sequence, for the first module to perform back propagation, that is, the module arranged at the end of the model, the forward propagation result can be directly used as the input of the back propagation of the module, thereby realizing back propagation for the module.
[0063] Based on any of the above embodiments, in step 120, the performing back propagation on the target module includes: In the case that the target module is not the first module among the multiple modules to perform back propagation, the data obtained from the previous communication process is used as the back propagation input to perform back propagation on the target module.
[0064] Specifically, in the process of performing back propagation for the model, that is, in the process of performing back propagation for each module in the model in sequence, after the back propagation of the first module performing back propagation is completed, each subsequent module performing back propagation can use the data obtained after the communication processing performed before its own back propagation as the input for its own back propagation, thereby realizing back propagation for each subsequent module.
[0065] Based on any of the above embodiments, in step 140, determining a new target module from the multiple modules includes: Based on the execution order of back propagation of each module in the model, a next module executed after the target module is determined from the multiple modules as a new target module.
[0066] Specifically, the execution order of the back propagation of each module in the model may be the opposite order of the execution order of the forward propagation of each module in the model. For example, in the forward propagation, the execution order of each module in the model is module A→module B→module C→module D, then in the back propagation, the execution order of each module in the model is module D→module C→module B→module A.
[0067] After determining the execution order of back propagation of each module in the model, back propagation can be performed module by module. Specifically, after a module completes back propagation, communication processing and reverse gradient accumulation are performed on the back propagation result of the module, and after the communication processing and reverse gradient accumulation are completed, based on the execution order of back propagation of each module, the module that performs back propagation after the module is determined as the new target module for back propagation. In this way, in the process of sequential back propagation of each module in the model, concurrent communication processing and reverse gradient accumulation for the back propagation results of the module are interspersed, so that the time consumption of gradient accumulation can be covered by the time consumption of communication processing, thereby improving the model training performance.
[0068] Based on any of the above embodiments, Figure 2 This is the second schematic diagram of the model training method provided by the present invention. Figure 2 In the figure on the left, the model training method in the related technology is strictly executed in series according to the four stages of forward network, reverse network, gradient accumulation, and optimizer parameter update. Among them, the forward network means forward propagation for the model, and the reverse network means reverse propagation for the model. In this method, each stage is executed after the previous stage is completed, so there will be no timing disorder during the training process. However, since there is no timing overlap between the stages, the training process has disadvantages such as high peak memory and poor performance.
[0069] exist Figure 2In the figure on the right, the model training method provided by the present invention is shown. In the model training method provided by the present invention, considering the fact that there is a dependency between gradient accumulation and back propagation, but the dependency is not strong, the reverse network is segmented, and after the execution of part of the reverse network, the reverse communication operator and part of the reverse gradient accumulation are performed in parallel, so that the communication time is used to cover the calculation time, and the peak video memory is reduced at the same time.
[0070] Specifically, in the training process for the model, the forward network stage is first executed, that is, forward propagation is performed for the model. Subsequently, for each module after the model is segmented, the partial reverse network is sequentially executed, that is, reverse propagation is performed sequentially for each module, and after each partial reverse network execution is completed, the reverse communication operator and partial reverse gradient accumulation are executed concurrently, wherein the reverse communication operator is used to perform communication processing on the reverse propagation result of the module, and the partial reverse gradient accumulation is used to perform gradient accumulation on the gradient in the reverse propagation result of the module. It can be understood that the gradient accumulation here is the accumulation of the gradient in the reverse propagation result of the module, rather than the traditional accumulation of the gradient in the reverse propagation result of the entire model, that is, in units of modules rather than models, and the gradient accumulation operations of each module are independent of each other, so it can be understood as partial reverse gradient accumulation. After each partial reverse network is executed, the optimizer parameter update stage can be executed.
[0071] It can be understood that, different from the model training method in the related art, the model training method provided by the present invention divides the reverse network that was originally executed as a whole into multiple parts, and executes each part sequentially, and also divides the gradient accumulation that was originally executed after the overall reverse network. After each partial reverse network is executed, partial reverse gradient accumulation is performed in parallel with the reverse communication operator. As a result, the gradient accumulation that originally required a separate period of time in the related art can be dispersed in the reverse network execution process, and the time consumption of gradient accumulation is covered by the communication time consumption of the reverse communication operator in the reverse network execution. Therefore, compared with the model training method in the related art, the model training method provided by the present invention is shorter in time and has better performance.
[0072] Moreover, in the model training method provided by the present invention, after each completion of partial reverse gradient accumulation, the low-precision gradient temporary memory can be released, thereby alleviating the memory pressure during the training process.
[0073] Based on any of the above embodiments, Figure 3 This is the third flow chart of the model training method provided by the present invention. Figure 3As shown, the upper figure represents the model training method in the related art, where stream0 and stream1 represent the data stream for calculation and the data stream for communication, respectively. In the upper figure, Mlp_bwd is the back propagation of the Mlp module in the model. After Mlp_bwd is completed, the communication processing Reduce_scatter is executed. After Reduce_scatter is completed, Rmsnorm_bwd is executed, that is, the back propagation of the Rmsnorm module in the model is executed. After the back propagation of all modules is completed, the gradient accumulation is started, including the gradient accumulation Mlp_grad_acc for the Mlp module.
[0074] It can be understood that in the related art, while the data stream stream1 for communication performs the communication process Reduce_scatter, the data stream stream0 for calculation does not perform any calculation operation.
[0075] The figure below shows the model training method in the present invention. In the figure below, after Mlp_bwd is completed, data stream stream0 will execute the gradient accumulation Mlp_grad_acc for the Mlp module in advance, and data stream stream1 will execute the communication processing Reduce_scatter. That is, in terms of timing, the gradient accumulation Mlp_grad_acc and the communication processing Reduce_scatter are executed concurrently, and the time consumption of the gradient accumulation is covered in the communication processing, that is, the overlap in timing is achieved.
[0076] The model training device provided by the present invention is described below. The model training device described below and the model training method described above can be referenced to each other.
[0077] Figure 4 Schematic diagram of the structure of the model training device provided by the present invention. Figure 4 As shown, the device comprises: A division unit 410, configured to divide the model into a plurality of modules based on the communication processing performed by the model in back propagation; A back propagation unit 420, configured to determine a target module from the multiple modules and perform back propagation on the target module; A concurrent unit 430, configured to simultaneously perform communication processing and reverse gradient accumulation based on the back-propagation result when the back-propagation result of the target module is obtained; A selection unit 440, configured to return to determine a new target module from the multiple modules after the communication processing and the reverse gradient accumulation are completed, until the multiple modules complete reverse propagation; The iteration unit 450 is used to iterate the parameters of the model based on the results of the reverse gradient accumulation of the multiple modules.
[0078] In the device provided by the embodiment of the present invention, based on the communication processing performed by the model during back propagation, the model is divided into multiple modules, and communication processing and reverse gradient accumulation are performed synchronously after the back propagation of each module is completed, thereby masking the time consumption of reverse gradient accumulation in model training in the communication processing time consumption, thereby shortening the model training time consumption, improving the model training efficiency, and thus improving the model training performance.
[0079] Based on any of the above embodiments, the division unit is specifically used for: The model is divided into a plurality of modules by taking the starting position of the model for communication processing in the back propagation as the segmentation position of the modules in the model.
[0080] Based on any of the above embodiments, the concurrency unit is further used for: When it is determined that the reverse gradient accumulation is completed, the gradient in the back-propagation result is deleted from the memory of the artificial intelligence chip, and the artificial intelligence chip is used for the model training.
[0081] Based on any of the above embodiments, the device further includes a forward propagation unit, configured to: Performing forward propagation on the model to obtain a forward propagation result of the model; The reverse propagation unit specifically includes: In the case that the target module is the first module among the multiple modules to perform back propagation, the forward propagation result is used as the back propagation input to perform back propagation on the target module.
[0082] Based on any of the above embodiments, the reverse propagation unit is specifically used for: In the case that the target module is not the first module among the multiple modules to perform back propagation, the data obtained from the previous communication process is used as the back propagation input to perform back propagation on the target module.
[0083] Based on any of the above embodiments, the selection unit is specifically used for: Based on the execution order of back propagation of each module in the model, a next module executed after the target module is determined from the multiple modules as a new target module.
[0084] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the model training method, which includes: Dividing the model into a plurality of modules based on the communication processing performed by the model during back propagation; Determine a target module from the multiple modules, and perform back propagation on the target module; When a back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result; After the communication processing and the reverse gradient accumulation are completed, returning to determine a new target module from the multiple modules until the multiple modules complete reverse propagation; Based on the result of the reverse gradient accumulation, parameter iteration is performed on the model.
[0085] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0086] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the model training method provided by the above methods, the method includes: Dividing the model into a plurality of modules based on the communication processing performed by the model during back propagation; Determine a target module from the multiple modules, and perform back propagation on the target module; When a back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result; After the communication processing and the reverse gradient accumulation are completed, returning to determine a new target module from the multiple modules until the multiple modules complete reverse propagation; Based on the result of the reverse gradient accumulation, parameter iteration is performed on the model.
[0087] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model training method provided by the above methods, the method comprising: Dividing the model into a plurality of modules based on the communication processing performed by the model during back propagation; Determine a target module from the multiple modules, and perform back propagation on the target module; When a back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result; After the communication processing and the reverse gradient accumulation are completed, returning to determine a new target module from the multiple modules until the multiple modules complete reverse propagation; Based on the result of the reverse gradient accumulation, parameter iteration is performed on the model.
[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model training method, characterized in that: include: Dividing the model into a plurality of modules based on the communication processing performed by the model during back propagation; Determine a target module from the multiple modules, and perform back propagation on the target module; When a back-propagation result of the target module is obtained, communication processing and reverse gradient accumulation are performed simultaneously based on the back-propagation result; After the communication processing and the reverse gradient accumulation are completed, returning to determine a new target module from the multiple modules until the multiple modules complete reverse propagation; Based on the result of the reverse gradient accumulation, parameter iteration is performed on the model.
2. The model training method according to claim 1, characterized in that: The communication processing performed by the model in the back propagation divides the model into multiple modules, including: The model is divided into a plurality of modules by taking the starting position of the model for communication processing in the back propagation as the segmentation position of the modules in the model.
3. The model training method according to claim 1, characterized in that: Also includes: When it is determined that the reverse gradient accumulation is completed, the gradient in the back-propagation result is deleted from the memory of the artificial intelligence chip, and the artificial intelligence chip is used for the model training.
4. The model training method according to claim 1, characterized in that: Also includes: Performing forward propagation on the model to obtain a forward propagation result of the model; The performing back propagation on the target module includes: In the case that the target module is the first module among the multiple modules to perform back propagation, the forward propagation result is used as the back propagation input to perform back propagation on the target module.
5. The model training method according to claim 1, characterized in that: The performing back propagation on the target module includes: In the case that the target module is not the first module among the multiple modules to perform back propagation, the data obtained from the previous communication process is used as the back propagation input to perform back propagation on the target module.
6. The model training method according to any one of claims 1 to 5, characterized in that: The determining a new target module from the multiple modules comprises: Based on the execution order of back propagation of each module in the model, a next module executed after the target module is determined from the multiple modules as a new target module.
7. A model training device, characterized in that: include: A partitioning unit, for partitioning the model into a plurality of modules based on communication processing performed by the model in back propagation; A back propagation unit, used to determine a target module from the multiple modules and perform back propagation on the target module; A concurrent unit, configured to simultaneously perform communication processing and reverse gradient accumulation based on the back-propagation result when a back-propagation result of the target module is obtained; A selection unit, configured to return to determine a new target module from the multiple modules after the communication processing and the reverse gradient accumulation are completed, until the multiple modules complete reverse propagation; An iteration unit is used to iterate the parameters of the model based on the results of reverse gradient accumulation of the multiple modules.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the model training method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6.