Model training method based on distributed data parallelism, model training method and related equipment

Abstract

The embodiment of the invention discloses a distributed data parallelism-based model training method, a model training method and related equipment, which are used for improving the model training efficiency. In a distributed data parallel scene, each training end participating in training only needs to synchronize the weighted training loss of each round, and adjusts a local model deployed on the training end based on the weighted training loss obtained through synchronization. Wherein the weighted training loss of each round is a weighted average value of the training loss of the corresponding round of the plurality of training ends participating in the training. Besides, if a plurality of training ends share at least part of the network layer of the large model, each training end only needs to be synchronized at two positions: 1, the scene is similar to the previous scene, and under the condition that the plurality of training ends do not share the last layer of network, the training ends are synchronized; when each instance of the last layer of network is operated, the weighted training loss needs to be subjected to independent back propagation and parameter updating; and 2, performing aggregation averaging of the reverse backhaul gradient on a single instance shared by a plurality of training ends when parameter updating is performed in the reverse propagation.

Description

[0001] This application claims the priority of Chinese patent application with application number 202410650013.7 filed on May 23, 2024, and invention name “Model training method and related equipment based on distributed data parallelism”, and the priority of Chinese patent application with application number 202411998647.8 filed on December 31, 2024, and invention name “Model training method and related equipment”. The entire contents of these two patent applications are incorporated into this application by reference. Technical Field

[0002] The embodiments of the present application relate to the field of model training, and in particular to a model training method based on distributed data parallelism, a model training method, and related equipment. Background Art

[0003] Deep learning models are profoundly and dramatically changing all aspects of today's society, work, and life. However, as the model size (parameters) grows, it is no longer possible to train the model on a single computing device (such as a graphics card) or a single computing node. To solve this problem, the industry and academia have proposed a variety of parallel and distributed algorithms, among which the most representative algorithms are distributed data parallelism, distributed model parallelism (pipeline parallelism, tensor parallelism), etc.

[0004] In fact, even with existing technical solutions, the cost of training a large model is extremely high. For a large model with billions of parameters, the training cost, including hardware investment and daily operation and maintenance, is usually as high as tens of millions or even hundreds of millions of dollars. Therefore, how to improve the training efficiency of large models has become a major problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a model training method based on distributed data parallelism, a model training method and related equipment to improve the efficiency of model training.

[0006] A first aspect of an embodiment of the present application provides a model training method based on distributed data parallelism, which is applied to a current training end and includes:

[0007] Obtaining the Nth round of training data of the current training end;

[0008] Based on the Nth round of training data of the current training end, training the local model of the current training end, and obtaining the Nth round of training loss of the current training end;

[0009] Obtain the weighted training loss of the Nth round, where the weighted training loss is the weighted average of the training losses of the Nth round of multiple training terminals, and the multiple training terminals include the current training terminal. The training loss of the Nth round of each training terminal is determined based on a preset loss function and the forward propagation result of the training data of the Nth round of the training terminal in the local model of the training terminal. When N equals 1, the local model of the training terminal is the initial model; when N is greater than 1, the local model of the training terminal is the local model of the training terminal after the (N - 1)th round of training;

[0010] Adjust the local model of the current training terminal based on the weighted training loss of the Nth round to obtain a new local model of the current training terminal after the Nth round of training, and end the training until the target model is obtained.

[0011] In a specific implementation manner, the obtaining of the weighted training loss of the Nth round includes:

[0012] Send the training loss of the Nth round of the current training terminal to the first aggregation terminal, and receive the weighted training loss of the Nth round sent by the first aggregation terminal. The weighted training loss of the Nth round is obtained by the first aggregation terminal performing a weighted average process on the training losses of the Nth round of the multiple training terminals;

[0013] Or,

[0014] Receive the training loss of the Nth round sent by each other training terminal, where the other training terminals are the training terminals other than the current training terminal among the multiple training terminals; perform a weighted average process on the training losses of the Nth round of the multiple training terminals to obtain the weighted training loss of the Nth round.

[0015] In a specific implementation manner, before the target model ends training, the method further includes:

[0016] If the current training terminal meets the preset aggregation condition, send the local model parameters of the local model of the current training terminal to the second aggregation terminal;

[0017] Overwrite the local model parameters based on the aggregated model parameters sent by the second aggregation terminal to obtain a new local model of the current training terminal. The aggregated model parameters are obtained by the second aggregation terminal performing a weighted average process on the local model parameters sent by each training terminal.

[0018] The second aspect of the embodiments of the present application provides a model training method based on distributed data parallelism, which is applied to an aggregation terminal and includes:

[0019] Receive the Nth round of training losses sent by each training end, where the Nth round of training loss of each training end is obtained by training the local model of the training end based on the Nth round of training data of the training end. When N equals 1, the local model of the training end is the initial model; when N is greater than 1, the local model of the training end is the new local model after the (N - 1)th round of training of the training end.

[0020] Perform weighted average processing on the Nth round of training losses sent by each training end to obtain the Nth round of weighted training loss.

[0021] Send the target training loss to each training end so that each training end adjusts its local model based on the Nth round of target training loss until the target model is obtained and the training ends.

[0022] In a specific implementation manner, the method further includes:

[0023] Receive the local model parameters of the local models sent by each training end.

[0024] Perform weighted average processing on the local model parameters sent by each training end to obtain aggregated model parameters.

[0025] Send the aggregated model parameters to each training end so that each training end overwrites the model parameters of its local model based on the aggregated model parameters to obtain a new local model.

[0026] A third aspect of the embodiments of the present application provides a current training end, including:

[0027] A first acquisition unit, configured to acquire the Nth round of training data of the current training end.

[0028] A training unit, configured to train the local model of the current training end based on the Nth round of training data of the current training end and obtain the Nth round of training loss of the current training end.

[0029] A second acquisition unit, configured to acquire the Nth round of weighted training loss, where the weighted training loss is the weighted average of the Nth round of training losses of multiple training ends, and the multiple training ends include the current training end. The Nth round of training loss of each training end is determined based on a preset loss function and the forward propagation result of the Nth round of training data of the training end in the local model of the training end. When N equals 1, the local model of the training end is the initial model; when N is greater than 1, the local model of the training end is the local model after the (N - 1)th round of training of the training end.

[0030] The training unit is further configured to adjust the local model of the current training end based on the weighted training loss of the Nth round, and obtain a new local model of the current training end after the Nth round of training, until a target model is obtained and the training ends.

[0031] In a specific implementation manner, the second obtaining unit is specifically configured to send the training loss of the Nth round of the current training end to the first aggregation end, and receive the weighted training loss of the Nth round sent by the first aggregation end, where the weighted training loss of the Nth round is obtained by the first aggregation end performing a weighted average process on the training losses of the Nth round of the multiple training ends;

[0032] Or,

[0033] Receive the training loss of the Nth round sent by each other training end, where the other training ends are the training ends other than the current training end among the multiple training ends; perform a weighted average process on the training losses of the Nth round of the multiple training ends to obtain the weighted training loss of the Nth round.

[0034] In a specific implementation manner, before the target model ends training, the training unit is further configured to, if the current training end meets a preset aggregation condition, send the local model parameters of the local model of the current training end to the second aggregation end;

[0035] Overwrite the local model parameters based on the aggregated model parameters sent by the second aggregation end to obtain a new local model of the current training end, where the aggregated model parameters are obtained by the second aggregation end performing a weighted average process on the local model parameters sent by each training end.

[0036] A fourth aspect of the embodiments of the present application provides an aggregation end, including:

[0037] A receiving unit, configured to receive the training loss of the Nth round sent by each training end, where the training loss of the Nth round of each training end is obtained by training the local model of the training end based on the training data of the Nth round of the training end, where when N is equal to 1, the local model of the training end is an initial model, and when N is greater than 1, the local model of the training end is the new local model of the training end after the (N - 1)th round of training;

[0038] A weighting unit, configured to perform a weighted average process on the training losses of the Nth round sent by each training end to obtain a weighted training loss of the Nth round;

[0039] A sending unit, configured to send a target training loss to each training end, so that each training end adjusts its own local model based on the target training loss of the Nth round until a target model is obtained and the training ends.

[0040] In a specific implementation manner, the receiving unit is further configured to receive the local model parameters of the local models sent by each of the training ends;

[0041] The weighting unit is further configured to perform weighted average processing on the local model parameters sent by each of the training ends to obtain aggregated model parameters;

[0042] The sending unit is further configured to send the aggregated model parameters to each of the training ends, so that each of the training ends overwrites the model parameters of its own local model based on the aggregated model parameters to obtain a new local model.

[0043] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: obtaining the Nth round of training data of the current training end; and training the local model of the current training end based on the Nth round of training data of the current training end to obtain the Nth round of training loss of the current training end. At the same time, in the scenario of distributed data parallelism, multiple training ends usually including the current training end start training from the initial model at the same time. The present application uses the weighted average value of the Nth round of training losses of multiple training ends as the Nth round of weighted training loss, and adjusts the local model of the current training end based on the Nth round of weighted training loss to obtain a new local model of the current training end after the Nth round of training until the target model is obtained and the training ends. In the above training process, only one piece of data, that is, the training loss, needs to be sent during each training synchronization. Specifically, when the forward propagation of each training end is completed, the training losses of the forward propagations of multiple training ends are aggregated and averaged, and the weighted training loss after the aggregation and averaging is used as the loss value of each training end for independent backpropagation and parameter update. Compared with the prior art that needs to transmit the model parameters or gradients of each layer of each training end (usually including hundreds of millions of data) during each backpropagation process, the embodiments of the present application have better training synchronization efficiency. While ensuring the training accuracy of the model, it greatly reduces the network data transmission volume during data parallel training, reduces the data transmission requirements and data waiting time for the acceleration graphics card, thereby greatly reducing the model training time and greatly improving the model training efficiency.

[0044] A fifth aspect of the embodiments of the present application provides a model training method, which is applied to a large model including M layers of networks, and each layer of network runs in one or L instances, where M is greater than 1 and L is greater than 1. The method further includes:

[0045] Obtaining the training loss of the large model in the Nth round of training;

[0046] If the number of instances in the j-th layer is 1 and the number of instances in the (j + 1)-th layer is L, then when performing parameter update during backpropagation, the mean of the backpropagated reverse gradients of the (j + 1)-th layer network of multiple instances in the (j + 1)-th layer during the N-th round of training is used as the aggregated average gradient of the (j + 1)-th layer network during the N-th round of training, and the product of the aggregated average gradient of the (j + 1)-th layer network during the N-th round of training and the Jacobian matrix of the j-th layer network of the j-th instance is used to calculate the parameter gradient of the j-th layer network of the j-th instance. The backpropagated reverse gradients of the (j + 1)-th layer network of multiple instances in the (j + 1)-th layer during the N-th round of training are: the gradient of the training loss with respect to the input of the (j + 1)-th layer network of the (j + 1)-th instance during the forward propagation of the N-th round of training;

[0047] Update the j-th layer network of the j-th instance based on the parameter gradient of the j-th layer network of the j-th instance.

[0048] In a specific implementation manner, the method further includes:

[0049] If the number of instances in the j-th layer is L and the number of instances in the (j + 1)-th layer is 1, then for each j-th layer network of the j-th instance, the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th instance during the N-th round of training and the Jacobian matrix of the j-th layer network of the j-th instance is used as the parameter gradient of the j-th layer network of the j-th instance.

[0050] In a specific implementation manner, the method further includes:

[0051] If the number of instances in the j-th layer is equal to the number of instances in the (j + 1)-th layer, then for each j-th layer network of the j-th instance, the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th instance corresponding to the j-th instance during the N-th round of training and the Jacobian matrix of the j-th layer network of the j-th instance is used as the parameter gradient of the j-th layer network of the j-th instance.

[0052] In a specific implementation manner, the method further includes:

[0053] If the large model meets the preset parameter synchronization condition, then each layer network running in the L instances is determined as a layer to be synchronized;

[0054] For each layer to be synchronized, obtain L copies of the parameters to be synchronized corresponding to the layer to be synchronized from the L instances running the layer to be synchronized, and based on the L copies of the parameters to be synchronized corresponding to the layer to be synchronized, determine the aggregated parameters corresponding to the layer to be synchronized, and update the parameters to be synchronized in the L instances running the layer to be synchronized to the aggregated parameters corresponding to the layer to be synchronized.

[0055] In a specific implementation, applied to the current training end, the large model is trained in parallel by L training ends, the L training ends include the current training end, and at least one layer of the network in the large model is shared by the L training ends. Each layer of the network shared by the L training ends runs in one instance. If the M-th layer of the network of the large model runs in L instances, then obtaining the training loss of the large model in the N-th round of training includes:

[0056] Obtain the N-th weighted training loss as the training loss of the large model in the N-th round. The N-th weighted training loss is the weighted average of the N-th round training losses of the L training ends, and the L training ends include the current training end. The N-th round training loss of each training end is determined based on a preset loss function and the output of the M-th layer of the network running on the training end in the N-th round of training.

[0057] In a specific implementation, the N-th weighted training loss is calculated by the current training end based on the N-th round training losses of each training end, or the N-th weighted training loss is calculated by an aggregation end based on the N-th round training losses of each training end.

[0058] A sixth aspect of the embodiments of the present application provides a training end, characterized in that the training end is used to train a large model including M layers of networks, and each layer of the network runs in one or L instances, where L>1, and includes:

[0059] A third obtaining unit, configured to obtain the training loss of the large model in the N-th round of training;

[0060] A calculation unit, configured to, if the number of instances of the j-th layer is 1 and the number of instances of the j+1-th layer is L, then when performing parameter update in backpropagation, use the mean of the backpropagated reverse gradients of the j+1-th layer of the network of multiple j+1-th layer instances in the N-th round of training as the aggregated average gradient of the j+1-th layer of the network in the N-th round of training, and calculate the parameter gradient of the j-th layer of the network of the j-th layer instance using the product of the aggregated average gradient of the j+1-th layer of the network in the N-th round of training and the Jacobian matrix of the j-th layer of the network of the j-th layer instance. The backpropagated reverse gradients of the j+1-th layer of the network of multiple j+1-th layer instances in the N-th round of training are: the gradient of the training loss with respect to the input of the j+1-th layer of the network of the j+1-th layer instance during the forward propagation of the N-th round of training;

[0061] An update unit, configured to update the j-th layer of the network of the j-th layer instance based on the parameter gradient of the j-th layer of the network of the j-th layer instance.

[0062] In a specific implementation, the computing unit is further configured to, if the number of instances in the j-th layer is L and the number of instances in the (j + 1)-th layer is 1, then for the j-th layer network of each j-th layer instance, use the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th layer instance and the Jacobian matrix of the j-th layer network of the j-th layer instance in the N-th round of training as the parameter gradient of the j-th layer network of the j-th layer instance.

[0063] In a specific implementation, the computing unit is further configured to, if the number of instances in the j-th layer is equal to the number of instances in the (j + 1)-th layer, then for the j-th layer network of each j-th layer instance, use the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th layer instance corresponding to the j-th layer instance one by one and the Jacobian matrix of the j-th layer network of the j-th layer instance in the N-th round of training as the parameter gradient of the j-th layer network of the j-th layer instance.

[0064] In a specific implementation, the training end further includes: a determination unit;

[0065] The determination unit is configured to, if the large model meets a preset parameter synchronization condition, then determine each layer network running in the L instances as a layer to be synchronized;

[0066] The determination unit is further configured to, for each layer to be synchronized, obtain L copies of parameters to be synchronized corresponding to the layer to be synchronized from the L instances running the layer to be synchronized, and based on the L copies of parameters to be synchronized corresponding to the layer to be synchronized, determine the aggregated parameters corresponding to the layer to be synchronized, and update the parameters to be synchronized in the L instances running the layer to be synchronized to the aggregated parameters corresponding to the layer to be synchronized.

[0067] In a specific implementation, the training end is the current training end, the large model is trained in parallel by L training ends, the L training ends include the current training end, at least one layer network in the large model is shared by the L training ends, each layer network shared by the L training ends runs in one instance, if the M-th layer network of the large model runs in L instances, then the third obtaining unit is specifically configured to obtain the N-th round weighted training loss as the training loss of the large model in the N-th round of training, the N-th round weighted training loss is the weighted average of the N-th round training losses of the L training ends, the L training ends include the current training end, and the N-th round training loss of each training end is determined based on a preset loss function and the output of the M-th layer network run by the training end in the N-th round of training.

[0068] In a specific implementation manner, the Nth-round weighted training loss is calculated by the current training end based on the Nth-round training loss of each training end, or the Nth-round weighted training loss is calculated by the aggregation end based on the Nth-round training loss of each training end.

[0069] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages: The present application provides a new model training framework. In a scenario where multiple training ends externally appear as data parallel, if a certain network layer belongs to a network with large storage resource occupancy but small computing resource occupancy, then all training ends can share this network layer running in one instance. This reduces the excessive occupancy of storage resources when deploying multiple instances and can effectively improve resource utilization. In addition, when performing model training under the training framework of the embodiments of the present application, it is only necessary to perform aggregation averaging and backpropagation at two places, and only synchronization is required at these two places: First, when the forward propagation of each training end is completed, aggregate and average the training losses of the forward propagation of multiple training ends, and use the aggregated and averaged loss value as the loss value of each instance, and perform independent backpropagation and parameter update in the instances of different training ends; Second, when performing parameter update during backpropagation, aggregate and average the backpropagated gradients in a single instance shared by multiple training ends. Therefore, compared with the existing training framework, the embodiments of the present application greatly reduce the data transmission volume between multiple instances and the waiting time for synchronization between multiple instances, and at the same time greatly reduce the actual complexity of the system, and can effectively improve the training efficiency.

[0070] The seventh aspect of the embodiments of the present application provides a computer device, including:

[0071] A central processing unit, a memory, and an input / output interface;

[0072] The memory is a transient storage memory or a persistent storage memory;

[0073] The central processing unit is configured to communicate with the memory and execute the instruction operations in the memory to execute the methods described in the first aspect, the second aspect, or the fifth aspect.

[0074] The eighth aspect of the embodiments of the present application provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the methods described in the first aspect, the second aspect, or the fifth aspect.

[0075] The ninth aspect of the embodiments of the present application provides a computer storage medium, in which instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the methods described in the first aspect, the second aspect, or the fifth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A system architecture diagram of the model training method based on distributed data parallel disclosed in the embodiments of the present application;

[0077] Figure 2 A schematic flow diagram of the model training method based on distributed data parallel disclosed in the embodiments of the present application;

[0078] Figure 3 A schematic diagram of the training effect of the model training method based on distributed data parallel disclosed in the embodiments of the present application for convolutional networks;

[0079] Figure 4 A schematic diagram of the training effect of the model training method based on distributed data parallel disclosed in the embodiments of the present application for Transformer models;

[0080] Figure 5 A schematic structural diagram of the training end of the model training method based on distributed data parallel disclosed in the embodiments of the present application;

[0081] Figure 6 A schematic structural diagram of the aggregation end of the model training method based on distributed data parallel disclosed in the embodiments of the present application;

[0082] Figure 7 A schematic diagram of an existing training framework disclosed in the embodiments of the present application;

[0083] Figure 8 A schematic diagram of the training framework of the present application disclosed in the embodiments of the present application;

[0084] Figure 9 A schematic flow diagram of the model training method disclosed in the embodiments of the present application;

[0085] Figure 10 A schematic structural diagram of the training end in the model training method disclosed in the embodiments of the present application;

[0086] Figure 11 A schematic structural diagram of a computer device disclosed in the embodiments of the present application. Detailed implementation manners

[0087] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0088] Embodiments of the present application provide a model training method, a model training method, and related devices based on distributed data parallelism, which are used to improve the model training efficiency.

[0089] To better implement the model training method based on distributed data parallelism of the present application, embodiments of the present application provide a model training architecture as shown in Figure 1 which includes multiple training terminals. Embodiments of the present application are implemented based on a distributed data parallel architecture. Each training terminal participating in the training (such as the current training terminal and other training terminals) externally appears as a certain parallel unit of a complete distributed data parallel scheme. Further, in some specific implementation manners, the training terminal can be an implementation of a certain distributed model parallel scheme (pipeline parallelism, tensor parallelism) internally. In this case, the model training belongs to the implementation of hybrid parallelism. In embodiments of the present application, after each round of training, a weighted average process is performed on the training losses of each training terminal in this round of training to obtain the weighted training loss of this round; then each training terminal updates the parameters according to the weighted training loss of this round to obtain a new local model. If the new local model after the update meets the convergence condition, the training can be ended; if the new local model does not meet the convergence condition, the iterative training continues. The weighted average process can be processed by each training terminal itself, or an aggregation terminal can be introduced for processing, which is not limited here.

[0090] Among them, in the distributed data parallel architecture, the training data in the training dataset will be distributed to each training terminal participating in the training, and each training terminal will determine the training data for each round of training according to the distributed training data.

[0091] Please refer to Figure 2 , based on the foregoing model training architecture, the model training method based on distributed data parallelism of embodiments of the present application includes the following steps:

[0092] 201. Obtain the Nth round of training data of the current training terminal.

[0093] To better illustrate the technical solution of embodiments of the present application, embodiments of the present application refer to any training terminal participating in distributed data parallel training as the current training terminal, and refer to each training terminal participating in distributed data parallel training except the foregoing current training terminal as other training terminals. Next, embodiments of the present application will describe the model training method of embodiments of the present application from the perspective of the current training terminal. Considering that any training terminal deploying the methods of embodiments of the present application and related embodiments can achieve more efficient model training through embodiments of the present application and related embodiments, that is, any training terminal participating in distributed data parallel training can be used as the current training terminal of embodiments of the present application to deploy and run the methods of embodiments of the present application and related embodiments.

[0094] For large models with a large number of parameters, the existing technologies generally distribute the data in the entire training dataset to each training node. For example, directly send the training data of a specified round (or a specified mini-batch) to each training node, or uniformly send the training data of multiple rounds to each training node, and each training node selects the training data for each round by itself to improve the computational parallelism during the model training process.

[0095] Therefore, before the current training node performs the Nth round of training, it is necessary to obtain the Nth round of training data required for the Nth round of training.

[0096] 202. Based on the Nth round of training data of the current training node, train the local model of the current training node and obtain the Nth round of training loss of the current training node;

[0097] After obtaining the Nth round of training data, based on any required training method, input the Nth round of training data into the local model of the current training node for forward propagation, and obtain the Nth round of training loss of the current training node based on the forward propagation result. This step is similar to the existing method for obtaining the training loss and will not be elaborated here.

[0098] It should be noted that the general model training steps include: forward propagation, loss value calculation, backpropagation, and parameter update or model update. The training in this step is narrow-sense training, which only includes forward propagation and loss value calculation.

[0099] 203. Obtain the Nth round of weighted training loss, where the weighted training loss is the weighted average of the Nth round of training losses of multiple training nodes.

[0100] It can be understood that multiple training nodes include the current training node. The Nth round of training loss of each training node is determined based on a preset loss function and the forward propagation result of the Nth round of training data of the training node in its own local model. When N equals 1, the local model of the training node is the initial model, and when N is greater than 1, the local model of the training node is the new local model after the (N - 1)th round of training of the training node. In other words, before the end of the first round of training, the local models of each training node are the same initial model.

[0101] Among them, the weighted weight of each training node can be determined based on the data volume of its own Nth round of training data and is positively correlated with the data volume of its own Nth round of training data. No specific limitation is made here. It should be noted that if the weighted weights of each training node are the same, the weighted average processing is equivalent to the sum average processing.

[0102] For example, in the embodiment of the present application, two training terminals participate in the training. Among them, the current training terminal trains 60 samples in the Nth round, while the other training terminal trains 40 samples in the Nth round. If the weighted average is performed according to the data volume of the training data in the Nth round, the weighted training loss in the Nth round = 0.6 × the training loss of the current training terminal in the Nth round + 0.4 × the training loss of the other training terminal in the Nth round; if the sum average process is performed, the weighted training loss in the Nth round = 0.5 × the training loss of the current training terminal in the Nth round + 0.5 × the training loss of the other training terminal in the Nth round.

[0103] 204. Adjust the local model of the current training terminal based on the weighted training loss in the Nth round to obtain a new local model of the current training terminal after the Nth round of training, and end the training until the target model is obtained.

[0104] After obtaining the weighted training loss in the Nth round, perform backpropagation based on the weighted training loss in the Nth round, and update the parameters based on the results of the backpropagation to obtain a new local model until a local model that meets the requirements is obtained. Use the local model that meets the requirements as the target model and end the iterative training.

[0105] In the existing distributed data parallel training scheme, during the training process of each round of data, the gradients of the entire model in the training of this round are collected - averaged - returned multiple times or once through the existing Message Passing Interface MPI technology, such as the all - reduce technology. However, the scheme of the present application averages not the gradients of model training, but the loss function values (i.e., training losses) of the model during the forward propagation of the training data in this round. In the embodiment of the present application, only one data, the loss value, needs to be sent during each training synchronization process. Compared with the technical scheme of the existing technology that needs to transmit the complete model parameters or gradients (usually including hundreds of millions of data), the embodiment of the present application has better training synchronization efficiency and higher accuracy, and can effectively improve the data synchronization efficiency.

[0106] In addition to the above content, the embodiment of the present application specifically has the following several major technical effects:

[0107] 1. Greatly save time:

[0108] Due to the huge number of parameters of large models (up to tens of billions or even hundreds of billions), the amount of data to be transmitted is huge (2 * the number of workers * the number of parameters * the number of model optimizer gradient copies), and these data transmissions are carried out through network transmissions. And network transmission is the biggest performance bottleneck during model training. Compared with the existing scheme, the amount of data that needs to be transmitted in the scheme of the embodiment of the present application is one in tens of billions or hundreds of billions of the existing scheme, which can be ignored (2 * the number of workers * 4 * the number of model optimizer gradient copies), greatly improving the overall efficiency of large model training.

[0109] 2. The solution of the embodiment of the present application can greatly reduce the hardware investment:

[0110] a) In the existing distributed data parallel solution, due to the need to transmit a large amount of data in a short time, high-performance dedicated network transmission equipment and network transmission technologies are required, such as the RDMA network. However, in the solution of the embodiment of the present application, since the amount of transmitted data is small, the existing ordinary network transmission solution can be used.

[0111] b) In the existing solution, due to the huge calculation amount in the parameter averaging process, a separate parameter server is usually required. However, in the solution of the embodiment of the present application, since the calculation amount is small, an ordinary computing device can be used to complete it.

[0112] 3. The main idea of optimizing the existing distributed data parallel solution is the parallelism of operation and transmission, that is, when calculating the gradients of the Nth layer, the averaging of the gradients of the N+1th layer is synchronously performed. This solution requires synchronization for each layer of the local models at all training ends. Even when the hardware configurations are exactly the same, the operation times of all layers of the local models at all training ends cannot be exactly the same. Therefore, the setting of these synchronization points will cause the local models at each training end to be in a waiting state for each layer. However, in the solution of the embodiment of the present application, there is only one synchronization point for averaging the loss function values of each working model during the entire calculation process. Except for this, the local models at each training end run independently and are trained. Thus, the solution design is simplified and time is further saved.

[0113] 4. Greatly simplify the design, development and operation costs of the software system. In the existing solution, since it is necessary to consider the gradient transmission and synchronization of multiple operators in multiple training ends, the design is extremely complex, and the development and operation costs are extremely high. However, the solution of the embodiment of the present application only needs to be slightly modified on the ordinary solution and does not require a special software system.

[0114] 5. It is easier to integrate with other solutions: All existing large model solutions are without exception a mixture of distributed data parallel and a certain model parallel solution. Due to the complex design of the existing distributed data parallel solution, it is very difficult to integrate with other model parallel solutions. However, the solution of the embodiment of the present application can complete hybrid deployment with almost no modification to the model parallel solution.

[0115] In some specific ways, the current training end can obtain the weighted training loss of the Nth round in the following ways including but not limited to: sending the training loss of the Nth round of the current training end to the first aggregation end and receiving the weighted training loss of the Nth round sent by the first aggregation end, where the weighted training loss of the Nth round is obtained by the first aggregation end through weighted average processing of the training losses of the Nth round of multiple training ends; or, receiving the training losses of the Nth round sent by each other training end, where the other training ends are the training ends other than the current training end among the multiple training ends; and performing weighted average processing on the training losses of the Nth round of multiple training ends to obtain the weighted training loss of the Nth round.

[0116] Specifically, the difference lies in whether the weighted training loss in the Nth round is obtained by each training end independently performing weighted average processing on the training losses of multiple training ends in the Nth round, or by the first aggregator performing weighted average processing on the training losses of multiple training ends in the Nth round. Among them, if it is obtained by each training end independently, the current training end needs to receive the training losses of the Nth round sent by each other training end except the current training end among multiple training ends. Generally, since the training loss is extremely small (such as a few hundred bytes), therefore, whether it is obtained by each training end independently or by the first aggregator, the time spent will not be too much, and it can be selected according to needs.

[0117] Based on the foregoing embodiments, in some specific implementation manners, in order to improve the training efficiency of the initial model and obtain an accurate target model faster, the embodiments of the present application can also aggregate the model parameters of the local models of each training end under certain conditions to obtain aggregated model parameters, and use the aggregated model parameters to update the local models of each training end, including the following steps: If the current training end meets the preset aggregation condition, send the local model parameters of the local model of the current training end to the second aggregator; overwrite the local model parameters of the current training end based on the aggregated model parameters sent by the second aggregator to obtain a new local model of the current training end, and the aggregated model parameters are obtained by the second aggregator performing weighted average processing on the local model parameters sent by each training end.

[0118] Different from the training loss, the model parameters are usually extremely large. Therefore, if aggregated model parameters need to be obtained, it is necessary to perform model parameter aggregation through an aggregator (such as the second aggregator) to ensure the aggregation efficiency. Among them, the preset aggregation conditions include that the number of training rounds of each training end reaches the system aggregation round number, or the amount of training data used by each training end is the same, which is not limited here.

[0119] In addition to improving the training efficiency of the initial model and obtaining an accurate target model faster through the above methods, the present application can also test the local models of each training end through a test data set when any training end (such as the current training end or other training ends) meets the preset aggregation condition, and based on the model evaluation metrics of each model obtained from the test on the test data set, determine the local model with the best test performance based on the model evaluation metrics, and determine the local model parameters of the local model with the best test performance as the aggregated model parameters, so that each training end overwrites the model parameters of its local model with the aggregated model parameters.

[0120] It should be noted that both the first aggregator and the second aggregator in the present application belong to aggregators, and the two can be the same or different aggregators, which is not limited here.

[0121] The foregoing has described various implementation manners of the embodiments of the present application. The following shows the technical effects that can be achieved by the embodiments of the present application through specific data examples.

[0122] To better illustrate the effects of the model training method of the present application in the training processes of different networks or models, the embodiments of the present application adopt a distributed data parallel architecture, and respectively train a convolutional network and a Transformer model through the existing PyTorch solution and the model training method of the present application, to obtain the training data as Figure 3 and Figure 4 shown. It should be noted that since large models with billions of parameters require extremely high training costs to support, the networks or models trained in the embodiments of the present application have relatively few parameters, and only simulate multiple training ends through multiple processes on a computer, but this does not affect the embodiment of the technical solution of the present application. The technical effects are reflected, as detailed in the following analysis.

[0123] First, explain the table headers in Figure 4 from top to bottom. Rank: represents the number of the training end. In the embodiments of the present application, 0 and 1 represent two different processes; Epoch: represents the training cycle. After each epoch, the convolutional network will perform a complete learning on the entire training set; Average loss of the test set: the average loss value of the convolutional network trained in the embodiments of the present application on the test set. The lower the average loss value, the better the performance of the convolutional network; Accuracy: represents the accuracy of the trained convolutional network on the test set. The values in the brackets represent the specific number of correct classifications and the total number of classifications from left to right in sequence.

[0124] Figure 3 The data shown in Figure 3 is a comparison of the convergence speed and total training duration of the existing PyTorch solution and the model training method of the present application when training models on a convolutional network with a relatively small number of parameters (about 2,000 parameters). From the accuracies of the existing solution and the solution of the present application at different Epochs, it can be seen that the convergence speed of the solution of the present application is similar to that of the existing PyTorch solution; however, even in such a small convolutional network, the total training duration of the present application is much less than that of the existing solution. It should be noted that during the process of obtaining the training data shown in

[0125] Next, explain Figure 4Describe the headers of the tables below. For the first table,

Rank: Represents the number of the training end. In the embodiments of the present application, 0 and 1 represent two different processes; Epoch: Represents the training cycle. After each epoch, the Transformer model will perform a complete learning on the entire training set; Completed Batch / Total Batch: The number of completed batches and the total number of batches in the current training; Time Spent per Single Batch (seconds): The time required for a single batch of training (in seconds); Training Loss: The loss value of the model on the training set; Perplexity: An indicator used to measure the performance of the language model. The lower the perplexity, the better the model.

Validation Set Loss: The loss value of the trained model on the validation set; Validation Set Perplexity: The perplexity of the trained model on the validation set, reflecting the generalization performance of the model; Test Set Loss: The loss value of the trained model on the test set; Test Set Perplexity: The perplexity of the trained model on the test set.

[0126] Figure 4 The data shown in is a comparison of the language model training effects of the existing PyTorch solution and the model training method of the present application on the Transformer model and the PennTreeBank corpus respectively. It should be noted that in the process of obtaining Figure 3 the training data shown in, the model training method of the present application and the existing Pytorch solution use the same learning rate. However, during the model training process corresponding to the model training method of the present application, parameter synchronization is performed every 200 training rounds, and parameter periodic synchronization is introduced to obtain the aggregated model parameters for updating the model. From Figure 4 the table shown, it can be seen that the convergence speed of the model training method of the present application is similar to that of the existing PyTorch solution. However, even in such a small model (about 6,000 parameters), the training duration required for each batch of the model training method of the present application is much shorter than that required for each batch of the existing PyTorch solution. Although the training convergence speed of the model training method of the present application seems to be lower than that of the existing Pytorch solution, the accuracy of the target model obtained by the model training method of the present application on the test data set is higher than that of the existing Pytorch solution.

[0127] It can be understood that due to the present application Figure 3 、 4In related embodiments, multiple training ends are implemented using multiple processes. Therefore, the multiple training ends send their respective training losses to each other through inter-process communication, and then determine the weighted training loss for each round. Since the time required for inter-process communication is much less than the time required for network communication between different devices (such as servers or computer devices) in an actual training scenario, even in the scenario with a relatively small number of parameters in the embodiments of the present application, the present application can demonstrate the advantage in training efficiency. In the training of large models with billions of parameters, the advantage of the present application will be even more obvious.

[0128] The above examples show that the solution of the embodiments of the present application is applicable to all networks or all models, including convolutional networks for image processing and Transformer models for natural language processing; the solution of the embodiments of the present application is similar to the existing distributed data parallel solution in terms of training convergence speed and final training accuracy; the training time of the embodiments of the present application is much less than the existing distributed data parallel solution. Generally speaking, while maintaining a model performance comparable to the existing PyTorch solution, the model training method of the present application significantly improves the training efficiency and shortens the training time, and has high practical value.

[0129] Please refer to Figure 5 , the embodiments of the present application provide a training end, including:

[0130] A first acquisition unit 501, configured to acquire the Nth round of training data of the current training end;

[0131] A training unit 502, configured to train the local model of the current training end based on the Nth round of training data of the current training end, and obtain the Nth round of training loss of the current training end;

[0132] A second acquisition unit 503, configured to acquire the Nth round of weighted training loss, where the weighted training loss is the weighted average of the Nth round of training losses of multiple training ends, and the multiple training ends include the current training end. The Nth round of training loss of each training end is determined based on a preset loss function and the forward propagation result of the Nth round of training data of the training end in the local model of the training end. When N is equal to 1, the local model of the training end is the initial model, and when N is greater than 1, the local model of the training end is the local model of the training end after the (N - 1)th round of training;

[0133] The training unit 502 is further configured to adjust the local model of the current training end based on the Nth round of weighted training loss, and obtain a new local model of the current training end after the Nth round of training, until the target model is obtained and the training ends.

[0134] In a specific implementation manner, the second obtaining unit 503 is specifically configured to send the training loss of the Nth round of the current training end to the first aggregation end, and receive the weighted training loss of the Nth round sent by the first aggregation end, where the weighted training loss of the Nth round is obtained by the first aggregation end through weighted average processing of the training losses of the Nth round of multiple training ends;

[0135] Or,

[0136] Receive the training loss of the Nth round sent by each other training end, where the other training ends are the training ends other than the current training end among the multiple training ends; perform weighted average processing on the training losses of the Nth round of multiple training ends to obtain the weighted training loss of the Nth round.

[0137] In a specific implementation manner, before the training of the target model ends, the training unit 502 is further configured to, if the current training end meets the preset aggregation condition, send the local model parameters of the local model of the current training end to the second aggregation end;

[0138] Overwrite the local model parameters based on the aggregated model parameters sent by the second aggregation end to obtain a new local model of the current training end, where the aggregated model parameters are obtained by the second aggregation end through weighted average processing of the local model parameters sent by each training end.

[0139] Please refer to Figure 6 , an aggregation end provided by an embodiment of the present application includes:

[0140] A receiving unit 601, configured to receive the training loss of the Nth round sent by each training end, where the training loss of the Nth round of each training end is obtained by training the local model of the training end based on the training data of the Nth round of the training end, where when N is equal to 1, the local model of the training end is the initial model, and when N is greater than 1, the local model of the training end is the new local model after the (N - 1)th round of training of the training end;

[0141] A weighting unit 602, configured to perform weighted average processing on the training loss of the Nth round sent by each training end to obtain the weighted training loss of the Nth round;

[0142] A sending unit 603, configured to send the target training loss to each training end, so that each training end adjusts its own local model based on the target training loss of the Nth round until the target model is obtained and the training ends.

[0143] In a specific implementation manner, the receiving unit 601 is further configured to receive the local model parameters of the local model sent by each training end;

[0144] The weighting unit 602 is further configured to perform weighted average processing on the local model parameters sent by each training end to obtain the aggregated model parameters;

[0145] The sending unit 603 is further configured to send the aggregated model parameters to each training end, so that each training end overwrites the model parameters of its own local model based on the aggregated model parameters to obtain a new local model.

[0146] The loss backpropagation scheme adopted by the last layer network of the training end described in the foregoing embodiments can also be applied to the new model training framework provided in the embodiments of the present application. In the new model training framework, multiple training ends share at least one layer of the large model network, that is, each layer of the large model network runs in 1 or L instances. Next, the specific model training method under the new training framework of the embodiments of the present application, as well as the combination of the loss backpropagation scheme adopted by the last layer network in the foregoing embodiments and the new training framework of the present application, will be described.

[0147] Please refer to Figure 7 , in the existing distributed data parallel or hybrid parallel solutions, when performing parameter updates during backpropagation, it is necessary to aggregate and average the gradients in each layer of the network of multiple instances in distributed training and send them back to each instance. This implementation method not only has a huge amount of data transmission between multiple instances, but also requires parameter synchronization between multiple instances in each network layer of the model during each training iteration, increasing the waiting time caused by synchronization and greatly increasing the complexity of system design.

[0148] To solve the limitations brought by the existing training framework, based on the existing training framework, the present application provides a training framework as Figure 8 shown. When using the training framework of the embodiments of the present application for model training, multiple training ends in parallel will share some network layers in the large model (such as Figure 8 the first layer network, the second layer network, and the M-1 layer network shown), and these network layers are only deployed on a single instance. For other network layers except the foregoing partial network layers (such as Figure 8 the M-3 layer network, the M-2 layer network, and the M layer network shown), corresponding instances are deployed in each training end.

[0149] Specifically, the other network layers in each training end can be deployed in one or more instances, and the embodiments of the present application do not make specific limitations.

[0150] Please refer to Figure 9 , the present application provides a model training method, which is applied to a large model including M layers of networks, and each layer of the network runs in one or L instances, where L is greater than 1, and includes the following steps:

[0151] 901. Obtain the training loss of the large model in the Nth round of training.

[0152] In practical applications, large models undergo multiple rounds of training during the training process. After each round of training, it is necessary to calculate the parameter gradients through backpropagation and update the network parameters based on the calculated parameter gradients. To better illustrate the technical solution of the embodiments of the present application, the embodiments of the present application take the Nth round as an example to describe the specific implementation manner of the model training method in the Nth round of training process. In the following description, the traditional convention of existing literature is followed, and the model layer count is based on the count during forward propagation, that is, during forward propagation, the jth layer is calculated first and then the (j + 1)th layer. Conversely, during backpropagation, the gradient of the (j + 1)th layer is calculated first and the parameters of the (j + 1)th layer are updated, and then the gradient of the jth layer is calculated and the parameters of the jth layer are updated. In addition, when calculating the gradients of the network parameters of each layer, the forward gradient algorithm JVP and the backward gradient algorithm VJP can be used. In the following description, the backward gradient algorithm VJP, that is, VJ = P, is used to calculate the gradient of the jth layer. V is called the backpropagation reverse gradient from the (j + 1)th layer to the jth layer during forward propagation, J is called the parameter Jacobian matrix of the jth layer, and P is called the parameter gradient of the jth layer, which is simply referred to as the gradient of the jth layer in the existing literature. This P value is also the reverse backpropagation gradient passed back from the jth layer to the (j - 1)th layer.

[0153] Similarly, a large model contains multiple layers of networks. Therefore, the jth layer in the embodiments of the present application can be any layer in the M-layer network of the large model, and no specific limitation is made here.

[0154] 902. If the number of instances in the jth layer is 1 and the number of instances in the (j + 1)th layer is L, then when performing parameter update during backpropagation, the mean of the backpropagation reverse gradients of the (j + 1)th layer network of multiple (j + 1)th layer instances in the Nth round of training is used as the aggregated average gradient of the (j + 1)th layer network in the Nth round of training, and the product of the aggregated average gradient of the (j + 1)th layer network in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instances is used to calculate the parameter gradient of the jth layer network of the jth layer instances. The backpropagation reverse gradients of the (j + 1)th layer network of multiple (j + 1)th layer instances in the Nth round of training are: the gradient of the training loss with respect to the input of the (j + 1)th layer network of the (j + 1)th layer instances during the forward propagation of the Nth round of training. Among them, since the output of the jth layer network of the jth layer instances during forward propagation is the input of the (j + 1)th layer network of the (j + 1)th layer instances, therefore, the backpropagation reverse gradients of the (j + 1)th layer network of multiple (j + 1)th layer instances in the Nth round of training can also be considered as: the gradient of the training loss with respect to the output of the jth layer network of the jth layer instances during the forward propagation of the Nth round of training.

[0155] Specifically, in the embodiments of the present application, the only instance of the j-th layer is a single instance of the j-th layer network shared by multiple training terminals, and multiple instances of the (j + 1)-th layer are specific instances of multiple (j + 1)-th layer networks each owned by a single training terminal. This means that when performing parameter updates during backpropagation, it is necessary to perform an operation of aggregating and averaging the backpropagated reverse gradients of the (j + 1)-th layer networks of multiple instances of the (j + 1)-th layer to obtain the aggregated average gradient of the (j + 1)-th layer network. Then, the product of the aggregated average gradient of the (j + 1)-th layer network in the N-th round of training and the Jacobian matrix of the j-th layer network of this single instance of the j-th layer is used as the parameter gradient of the j-th layer network of this single instance of the j-th layer.

[0156] If j in the embodiments of the present application is Figure 8 M - 1 as shown in Figure 8 then the correspondence between the instance of the j-th layer and the instance of the (j + 1)-th layer in the embodiments of the present application is consistent with Figure 8 the correspondence between the instance of the (M - 1)-th layer and the instance of the M-th layer in Figure 8 Based on the arrow indication between the M-th layer network of each instance of the M-th layer and the unique (M - 1)-th layer network in Figure 8 it can be seen that the gradient of the M-th layer network obtained by the (M - 1)-th layer network is the aggregated average gradient, and this aggregated average gradient is the average of the sum of the backpropagated reverse gradients of the M-th layer networks of multiple instances of the M-th layer.

[0157] It should be noted that the backpropagated reverse gradient of any layer network is the gradient that needs to be backpropagated to its corresponding upper layer network during the process of gradient update in backpropagation; while the parameter gradient of any layer network is the gradient used when this layer network adjusts its own network parameters.

[0158] 903. Update the j-th layer network of the j-th layer instance based on the parameter gradient of the j-th layer network of the j-th layer instance.

[0159] Based on the foregoing embodiments, it can be known that the parameter gradient of any layer network is the gradient used when this layer network adjusts its own network parameters. Therefore, based on the parameter gradient of the j-th layer network of the j-th layer instance, new parameters of the j-th layer network of the j-th layer instance can be calculated. Finally, the parameters of the j-th layer network of the j-th layer instance are updated to the new parameters calculated above to complete the update of the j-th layer network of the j-th layer instance.

[0160] In the embodiments of the present application, in a scenario where multiple training terminals externally appear as data parallel, if a certain network layer requires a large amount of memory storage resources but relatively small computing resources, a scheme of sharing a single instance of this network layer among all training terminals can be adopted to reduce the excessive occupation of storage resources caused by each training terminal instance using a single instance of this network layer alone, effectively improving the utilization rate of memory storage resources and reducing the pressure on GPU memory resources.

[0161] In some specific implementations, if the number of instances in the j-th layer is L and the number of instances in the (j + 1)-th layer is 1, then for the j-th layer network of each j-th layer instance, the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th layer instance and the Jacobian matrix of the j-th layer network of the j-th layer instance in the N-th round of training is used as the parameter gradient of the j-th layer network of the j-th layer instance.

[0162] Specifically, in the embodiments of the present application, multiple j-th layer instances are specific instances of the j-th layer network separately owned by multiple training ends, and the only (j + 1)-th layer instance is a single instance of the (j + 1)-th layer network shared by multiple training ends. This means that when parameter updates are performed during backpropagation, the j-th layer network of each j-th layer instance calculates its own parameter gradient based on the product of the backpropagated reverse gradient of the only (j + 1)-th layer instance and its own Jacobian matrix. Equivalently, the backpropagated reverse gradient of the only (j + 1)-th layer instance is "copied" and sent to the j-th layer network of each j-th layer instance for calculating the parameter gradient of each j-th layer network.

[0163] If j in the embodiments of the present application is Figure 8 M - 2 as shown in Figure 8 then the correspondence between the j-th layer instance and the (j + 1)-th layer instance in the embodiments of the present application is the same as the correspondence between the (M - 2)-th layer instance and the (M - 1)-th layer instance in Figure 8 Based on the arrow indication between the (M - 2)-th layer network of each (M - 2)-th layer instance and the only (M - 1)-th layer network in

[0164] In some other specific implementations, if the number of instances in the j-th layer is equal to the number of instances in the (j + 1)-th layer, then for the j-th layer network of each j-th layer instance, the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the corresponding (j + 1)-th layer instance and the Jacobian matrix of the j-th layer network of the j-th layer instance in the N-th round of training is used as the parameter gradient of the j-th layer network of the j-th layer instance.

[0165] Specifically, if the number of instances in the j-th layer is equal to the number of instances in the (j + 1)-th layer, it means that each j-th layer instance has a corresponding (j + 1)-th layer instance. There are two possible cases as follows: 1. Multiple j-th layer instances are specific instances of multiple j-th layer networks separately owned by each training end, multiple (j + 1)-th layer instances are specific instances of multiple (j + 1)-th layer networks separately owned by each training end, and the j-th layer network of each instance is uniquely connected to the (j + 1)-th layer network of the same instance; 2. The only j-th layer instance is a single instance of the j-th layer network shared by multiple training ends, and the only (j + 1)-th layer instance is a single instance of the (j + 1)-th layer network shared by multiple training ends.

[0166] In the first case, this means that when parameter updates are performed during backpropagation, for each instance of the (j + 1)-th layer corresponding to a training end, the backpropagated reverse gradient of the (j + 1)-th layer network only needs to be passed back to the j-th layer network of the instance of the j-th layer corresponding to that training end. Then, for each instance of the j-th layer network of each training end, the parameter gradient of the j-th layer network is calculated based on the received backpropagated reverse gradient of the (j + 1)-th layer network of the corresponding instance of the (j + 1)-th layer. It should be noted that in the first case, in the embodiments of the present application, it is not necessary to aggregate and average the backpropagated reverse gradients of multiple (j + 1)-th layer networks, nor is it necessary to "copy" the backpropagated reverse gradients of the (j + 1)-th layer networks and send them to multiple j-th layer networks of the j-th layer instances.

[0167] If j in the embodiments of the present application is Figure 8 M - 3 as shown in Figure 8 then the correspondence between the j-th layer instance and the (j + 1)-th layer instance in the embodiments of the present application is the same as the correspondence between the (M - 3)-th layer instance and the (M - 2)-th layer instance in Figure 8 Based on the arrow indication between multiple (M - 3)-th layer networks of multiple (M - 3)-th layer instances and multiple (M - 2)-th layer networks in

[0168] In the second case, this means that when parameter updates are performed during backpropagation, the unique j-th layer network of the j-th layer instance calculates the parameter gradient based on the backpropagated reverse gradient of the unique (j + 1)-th layer network of the (j + 1)-th layer instance.

[0169] If j in the embodiments of the present application is Figure 8 the first layer network as shown in Figure 8 then the quantitative relationship between the j-th layer instance and the (j + 1)-th layer instance in the embodiments of the present application is the same as the correspondence between the first layer network instance and the second layer network instance in Figure 8 Based on the arrow indication between the unique first layer network of the unique first layer instance and the unique second layer network in

[0170] In some specific implementations, if the large model meets the preset parameter synchronization condition, each layer of the network running in L instances is determined as a layer to be synchronized; for each layer to be synchronized, L copies of the parameters to be synchronized corresponding to the layer to be synchronized are obtained from the L instances running the layer to be synchronized, and based on the L copies of the parameters to be synchronized corresponding to the layer to be synchronized, the aggregated parameters corresponding to the layer to be synchronized are determined, and the parameters to be synchronized in the L instances running the layer to be synchronized are updated to the aggregated parameters corresponding to the layer to be synchronized.

[0171] Based on the foregoing embodiments, in the case where multiple instances of the j-th layer are specific instances of multiple j-th layer networks separately owned by each training end, or multiple instances of the j + 1-th layer are specific instances of multiple j + 1-th layer networks separately owned by each training end, the embodiments of the present application do not synchronize the network parameters between multiple j + 1-th layer networks separately owned by different training ends.

[0172] Therefore, in order to further improve the convergence speed of the large model, on the basis of the foregoing embodiments, the embodiments of the present application also provide a technical solution to synchronize the network parameters between multiple layers to be synchronized separately owned by different training ends. Taking Figure 8 the large model shown as an example, Figure 8 the M - 3rd layer network, the M - 2nd layer network, and the Mth layer network shown in are all layers to be synchronized.

[0173] Specifically, for each layer to be synchronized network in the multi-layer network of the large model except for the network layers shared by multiple training ends, the embodiments of the present application will obtain L copies of the parameters to be synchronized corresponding to the layer to be synchronized from the L instances running the same layer to be synchronized. Among them, each copy of the L copies of the parameters to be synchronized: refers to all the parameters to be synchronized obtained from one instance running the corresponding layer to be synchronized. For example, Figure 8 if the M - 3rd layer network shown in is a layer to be synchronized, then all the parameters to be synchronized obtained from each M - 3rd layer instance are one copy of the parameters to be synchronized of the M - 3rd layer network.

[0174] Then, weighted aggregation processing is performed based on the L copies of the parameters to be synchronized corresponding to the layer to be synchronized to determine the aggregated parameters corresponding to the layer to be synchronized. Specifically, the weighted weights corresponding to the L instances running the same layer to be synchronized can be configured as needed. For example, the weighted weights corresponding to multiple instances can be the same or determined according to the number of training samples used by each instance in the Nth round of training, and no specific limitation is made here. Finally, the parameters to be synchronized in the L instances running the layer to be synchronized are updated to the aggregated parameters corresponding to the layer to be synchronized.

[0175] In practical applications, for the solution where the M-th layer network runs in L M-th layer instances, the training end that executes the method of the embodiment of the present application is the current training end, and the large model is trained in parallel by L training ends. The aforementioned L training ends include the current training end. The L training ends share at least one layer of the network in the large model, and each layer of the network shared by the L training ends runs in one instance. The calculation of the training loss in step 901 above can be specifically implemented in the following manner: Obtain the N-th round weighted training loss as the training loss of the large model in the N-th round of training. The N-th round weighted training loss is the weighted average of the N-th round training losses of the L training ends. The L training ends include the current training end. The N-th round training loss of each training end is determined based on a preset loss function and the output of the M-th layer network operated by the training end in the N-th round of training. Among them, the training loss of the large model in the N-th round of training in the model training method of the present application is the weighted training loss in the aforementioned model training method based on distributed data parallelism. The two can be calculated in a similar manner, which will not be elaborated here.

[0176] Specifically, the training loss of each training end in the N-th round of training is determined based on a preset loss function and the output of the M-th layer network operated by the training end in the N-th round of training, where the output of the M-th layer network operated by the training end in the N-th round of training is equivalent to the output of the M-th layer instance operated by the training end. The weighted training loss is the weighted average of the N-th round training losses of the L training ends. Among them, the weighted weight of each training end can be determined based on the data volume of its own N-th round of training data and is positively correlated with the data volume adopted by its own N-th round of training data. No specific limitation is made here. It should be noted that if the weighted weights of each training end are the same, the weighted average processing is equivalent to the summation average processing.

[0177] In addition, the calculation of the N-th round weighted training loss in the embodiment of the present application can be obtained by each training end participating in the training of the large model respectively receiving the N-th round training losses sent by the other L - 1 training ends (except itself among the L training ends), or by the corresponding aggregation end receiving the N-th round training losses of each training end and calculating them.

[0178] When performing model training under the training framework of the embodiments of the present application, it is only necessary to perform gradient aggregation averaging and backpropagation at two places, and only at these two places is synchronization required: First, when the forward propagation of each training end is completed, aggregate and average the training losses of the forward propagations of multiple training ends, and use the aggregated and averaged loss value as the loss value of each instance running the last layer of the network. Independent backpropagation and parameter update are performed among the instances of different training ends; Second, when performing parameter update during backpropagation, aggregate and average the backpropagated gradients of a single instance shared among multiple training ends. Therefore, compared with the existing training framework, the embodiments of the present application greatly reduce the data transmission volume between multiple instances and the waiting time for synchronization between multiple instances, and at the same time greatly reduce the actual complexity of the system, and can effectively improve the training efficiency.

[0179] Based on the foregoing embodiments, it can be seen that when L training ends share the last layer of the network (i.e., the Mth layer of the network), it is necessary to wait for all training ends to complete the forward propagation in sequence before the training loss of the Nth round of each training end can be obtained, and then calculate the weighted training loss of the Nth round.

[0180] Please refer to Figure 10 , the embodiments of the present application provide a training end, which is used to train a large model including an M-layer network, and each layer of the network runs in one or L instances, where L is greater than 1, including:

[0181] A third acquisition unit 1001, configured to acquire the training loss of the large model in the Nth round of training;

[0182] A calculation unit 1002, configured to, if the number of instances of the jth layer is 1 and the number of instances of the j+1th layer is L, when performing parameter update during backpropagation, use the mean of the backpropagated reverse gradients of the (j+1)th layer network of multiple (j+1)th layer instances in the Nth round of training as the aggregated average gradient of the (j+1)th layer network in the Nth round of training, and use the product of the aggregated average gradient of the (j+1)th layer network in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instance to calculate the parameter gradient of the jth layer network of the jth layer instance. The backpropagated reverse gradients of the (j+1)th layer network of multiple (j+1)th layer instances in the Nth round of training are: the gradient of the training loss with respect to the input of the (j+1)th layer network of the (j+1)th layer instance during the forward propagation of the Nth round of training;

[0183] An update unit 1003, configured to update the jth layer network of the jth layer instance based on the parameter gradient of the jth layer network of the jth layer instance.

[0184] In a specific implementation, the computing unit 1002 is further configured to, if the number of instances in the j-th layer is L and the number of instances in the (j + 1)-th layer is 1, then for the j-th layer network of each j-th layer instance, use the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th layer instance in the N-th round of training and the Jacobian matrix of the j-th layer network of the j-th layer instance as the parameter gradient of the j-th layer network of the j-th layer instance.

[0185] In a specific implementation, the computing unit 1002 is further configured to, if the number of instances in the j-th layer is equal to the number of instances in the (j + 1)-th layer, then for the j-th layer network of each j-th layer instance, use the product of the backpropagated reverse gradient of the (j + 1)-th layer network of the (j + 1)-th layer instance corresponding to the j-th layer instance in the N-th round of training and the Jacobian matrix of the j-th layer network of the j-th layer instance as the parameter gradient of the j-th layer network of the j-th layer instance.

[0186] In a specific implementation, the training end further includes: a determination unit;

[0187] The determination unit is configured to, if the large model meets the preset parameter synchronization condition, then determine each layer of the network running in the L instances as a layer to be synchronized;

[0188] The determination unit is further configured to, for each layer to be synchronized, obtain L copies of the parameters to be synchronized corresponding to the layer to be synchronized from the L instances running the layer to be synchronized, and based on the L copies of the parameters to be synchronized corresponding to the layer to be synchronized, determine the aggregated parameters corresponding to the layer to be synchronized, and update the parameters to be synchronized in the L instances running the layer to be synchronized to the aggregated parameters corresponding to the layer to be synchronized.

[0189] In a specific implementation, the training end is the current training end, the large model is trained in parallel by L training ends, the L training ends include the current training end, at least one layer of the network in the large model is shared by the L training ends, each layer of the network shared by the L training ends runs in one instance, if the M-th layer of the network of the large model runs in L instances, then the third acquisition unit 1001 is specifically configured to obtain the N-th round weighted training loss as the training loss of the large model in the N-th round of training, the N-th round weighted training loss is the weighted average of the N-th round training losses of the L training ends, the L training ends include the current training end, and the N-th round training loss of each training end is determined based on a preset loss function and the output of the M-th layer of the network run by the training end in the N-th round of training.

[0190] In a specific implementation, the N-th round weighted training loss is calculated by the current training end based on the N-th round training losses of each training end, or the N-th round weighted training loss is calculated by the aggregation end based on the N-th round training losses of each training end.

[0191] Figure 11It is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 1100 may include one or more central processing units (CPU) 1101 and a memory 1105. The memory 1105 stores one or more application programs or data.

[0192] The memory 1105 may be a volatile storage or a persistent storage. The program stored in the memory 1105 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit 1101 may be configured to communicate with the memory 1105 and execute a series of instruction operations in the memory 1105 on the computer device 1100.

[0193] The computer device 1100 may also include one or more power supplies 1102, one or more wired or wireless network interfaces 1103, one or more input and output interfaces 1104, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0194] The CPU 1101 can execute the aforementioned Figures 1 to 6 In the embodiment shown, the operations performed by the current training end or aggregation end, and / or the aforementioned Figures 7 to 10 The operations performed by the training end in the illustrated embodiment will not be described in detail here.

[0195] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the instructions of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0197] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0201] The embodiments of the present application also provide a computer program product containing instructions. When the computer program product runs on a computer, it causes the computer to execute the model training method and the model training method based on distributed data parallelism as described above.

Claims

1. A model training method based on distributed data parallelism, characterized in that, Applied to the current training end, including: Obtain the training data of the Nth round of the current training end; Based on the training data of the Nth round of the current training end, train the local model of the current training end, and obtain the training loss of the Nth round of the current training end; Obtain the weighted training loss of the Nth round. The weighted training loss is the weighted average of the training losses of the Nth round of multiple training ends, and the multiple training ends include the current training end. The training loss of the Nth round of each training end is determined based on a preset loss function and the forward propagation result of the training data of the Nth round of the training end in the local model of the training end. When N is equal to 1, the local model of the training end is the initial model. When N is greater than 1, the local model of the training end is the local model after the (N - 1)th round of training of the training end; Based on the weighted training loss of the Nth round, adjust the local model of the current training end to obtain a new local model of the current training end after the Nth round of training, until the target model is obtained and the training ends.

2. The model training method based on distributed data parallel according to claim 1, wherein The obtaining of the weighted training loss of the Nth round includes: Send the training loss of the Nth round of the current training end to the first aggregation end, and receive the weighted training loss of the Nth round sent by the first aggregation end. The weighted training loss of the Nth round is obtained by the first aggregation end through weighted average processing of the training losses of the Nth round of multiple training ends; Or, Receive the training loss of the Nth round sent by each other training end. The other training ends are the training ends other than the current training end among the multiple training ends; perform weighted average processing on the training losses of the Nth round of the multiple training ends to obtain the weighted training loss of the Nth round.

3. The model training method based on distributed data parallel according to claim 1 or 2, characterized in that Before the target model ends training, the method further includes: If the current training end meets the preset aggregation condition, send the local model parameters of the local model of the current training end to the second aggregation end; Overwrite the local model parameters based on the aggregation model parameters sent by the second aggregation end to obtain a new local model of the current training end. The aggregation model parameters are obtained by the second aggregation end through weighted average processing of the local model parameters sent by each training end.

4. A model training method based on distributed data parallelism, characterized in that, Applied to the aggregation end, including: Receive the training loss of the Nth round sent by each training end. The training loss of the Nth round of each training end is obtained by training the local model of the training end based on the training data of the Nth round of the training end. When N is equal to 1, the local model of the training end is the initial model. When N is greater than 1, the local model of the training end is the new local model after the (N - 1)th round of training of the training end; Perform weighted average processing on the training losses of the Nth round sent by each training end to obtain the weighted training loss of the Nth round; Send the target training loss to each training end, so that each training end adjusts its own local model based on the target training loss of the Nth round until the target model is obtained and the training ends.

5. The model training method based on distributed data parallelism according to claim 4, wherein The method further includes: Receive the local model parameters of the local model sent by each training end; Perform weighted average processing on the local model parameters sent by each training end to obtain the aggregation model parameters; Send the aggregated model parameters to each of the training terminals, so that each training terminal overwrites the model parameters of its own local model based on the aggregated model parameters to obtain a new local model.

6. A model training method, characterized in that, Applied to a large model including M layers of networks, each layer of network runs in one or L instances, M>1, L>1, the method further includes: Obtain the training loss of the large model in the Nth round of training; If the number of instances in the jth layer is 1 and the number of instances in the (j + 1)th layer is L, then when performing parameter update in backpropagation, use the mean of the backpropagated reverse gradients of the (j + 1)th layer network of multiple (j + 1)th layer instances in the Nth round of training as the aggregated average gradient of the (j + 1)th layer network in the Nth round of training, and use the product of the aggregated average gradient of the (j + 1)th layer network in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instance to calculate the parameter gradient of the jth layer network of the jth layer instance. The backpropagated reverse gradients of the (j + 1)th layer network of multiple (j + 1)th layer instances in the Nth round of training are: the gradient of the training loss with respect to the input of the (j + 1)th layer network of the (j + 1)th layer instance during the forward propagation of the Nth round of training; Update the jth layer network of the jth layer instance based on the parameter gradient of the jth layer network of the jth layer instance.

7. The model training method according to claim 6, wherein The method further includes: If the number of instances in the jth layer is L and the number of instances in the (j + 1)th layer is 1, then for each jth layer network of the jth layer instance, use the product of the backpropagated reverse gradient of the (j + 1)th layer network of the (j + 1)th layer instance in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instance as the parameter gradient of the jth layer network of the jth layer instance.

8. The model training method according to claim 6, wherein The method further includes: If the number of instances in the jth layer is equal to the number of instances in the (j + 1)th layer, then for each jth layer network of the jth layer instance, use the product of the backpropagated reverse gradient of the (j + 1)th layer network of the (j + 1)th layer instance corresponding to the jth layer instance in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instance as the parameter gradient of the jth layer network of the jth layer instance.

9. The model training method according to claim 6, wherein The method further includes: If the large model meets the preset parameter synchronization condition, then determine each layer of network running in L instances as a layer to be synchronized; For each layer to be synchronized, obtain L copies of to-be-synchronized parameters corresponding to the layer to be synchronized from the L instances running the layer to be synchronized, and based on the L copies of to-be-synchronized parameters corresponding to the layer to be synchronized, determine the aggregated parameters corresponding to the layer to be synchronized, and update the to-be-synchronized parameters in the L instances running the layer to be synchronized to the aggregated parameters corresponding to the layer to be synchronized.

10. The model training method according to any one of claims 6 to 9, characterized in that The training terminal is the current training terminal, the large model is trained in parallel by L training terminals, the L training terminals include the current training terminal, the L training terminals share at least one layer of network in the large model, each layer of network shared by the L training terminals runs in one instance. If the Mth layer network of the large model runs in L instances, then the obtaining the training loss of the large model in the Nth round of training includes: Obtain the weighted training loss of the Nth round as the training loss of the large model in the Nth round of training. The weighted training loss of the Nth round is the weighted average of the training losses of the Nth round of L training terminals, and the L training terminals include the current training terminal. The training loss of the Nth round of each training terminal is determined based on a preset loss function and the output of the Mth layer network run by the training terminal in the Nth round of training.

11. The model training method according to claim 10, wherein The weighted training loss of the Nth round is calculated by the current training terminal based on the training losses of the Nth round of each training terminal, or the weighted training loss of the Nth round is calculated by the aggregation terminal based on the training losses of the Nth round of each training terminal.

12. A training terminal, characterized in that, The training terminal is used to train a large model containing an M-layer network, and each layer of the network runs in one or L instances, where L>1, and includes: An acquisition unit, configured to acquire the training loss of the large model in the Nth round of training; A calculation unit, configured to, if the number of instances of the jth layer is 1 and the number of instances of the (j + 1)th layer is L, when performing parameter update in backpropagation, use the mean of the backpropagated reverse gradients of the (j + 1)th layer network of multiple instances of the (j + 1)th layer in the Nth round of training as the aggregated average gradient of the (j + 1)th layer network in the Nth round of training, and calculate the parameter gradient of the jth layer network of the jth layer instance using the product of the aggregated average gradient of the (j + 1)th layer network in the Nth round of training and the Jacobian matrix of the jth layer network of the jth layer instance. The backpropagated reverse gradients of the (j + 1)th layer network of multiple instances of the (j + 1)th layer in the Nth round of training are: the gradient of the training loss with respect to the input of the (j + 1)th layer network of the (j + 1)th layer instance during the forward propagation of the Nth round of training; An update unit, configured to update the jth layer network of the jth layer instance based on the parameter gradient of the jth layer network of the jth layer instance.

13. A current training terminal, characterized in that, Includes: A first acquisition unit, configured to acquire the training data of the Nth round of the current training terminal; A training unit, configured to train the local model of the current training terminal based on the training data of the Nth round of the current training terminal, and obtain the training loss of the Nth round of the current training terminal; A second acquisition unit, configured to acquire the weighted training loss of the Nth round. The weighted training loss is the weighted average of the training losses of the Nth round of multiple training terminals, and the multiple training terminals include the current training terminal. The training loss of the Nth round of each training terminal is determined based on a preset loss function and the forward propagation result of the training data of the Nth round of the training terminal in the local model of the training terminal. When N is equal to 1, the local model of the training terminal is the initial model, and when N is greater than 1, the local model of the training terminal is the local model of the training terminal after the (N - 1)th round of training; The training unit is further configured to adjust the local model of the current training terminal based on the weighted training loss of the Nth round to obtain a new local model of the current training terminal after the Nth round of training, and end the training until the target model is obtained.

14. An aggregation terminal, characterized in that, Includes: A receiving unit, configured to receive the Nth-round training loss sent by each training end, where the Nth-round training loss of each training end is obtained by training the local model of the training end based on the Nth-round training data of the training end. When N equals 1, the local model of the training end is an initial model. When N is greater than 1, the local model of the training end is the new local model of the training end after the (N - 1)th round of training; A weighting unit, configured to perform weighted average processing on the Nth-round training loss sent by each training end to obtain the Nth-round weighted training loss; A sending unit, configured to send a target training loss to each training end, so that each training end adjusts its own local model based on the Nth-round target training loss until a target model is obtained and the training ends.

15. A computer device, characterized in that, Comprising: A central processing unit, a memory, and an input / output interface; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute the instruction operations in the memory to execute the model training method based on distributed data parallelism described in any one of claims 1 to 3, the model training method based on distributed data parallelism described in any one of claims 4 to 5, and the model training method described in any one of claims 6 to 11.

16. A computer storage medium, characterized in that, Instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer is caused to execute the model training method based on distributed data parallelism described in any one of claims 1 to 3, the model training method based on distributed data parallelism described in any one of claims 4 to 5, and the model training method described in any one of claims 6 to 11.

17. A computer program product comprising instructions, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the model training method based on distributed data parallelism described in any one of claims 1 to 3, the model training method based on distributed data parallelism described in any one of claims 4 to 5, and the model training method described in any one of claims 6 to 11.

Images