Model parameter updating method and device, equipment, storage medium and program product

Abstract

The application relates to a model parameter updating method and device, computer equipment, a storage medium and a program product. The method relates to artificial intelligence and comprises the following steps: acquiring model state quantities of a network model to be updated; dividing the model state quantities into data parallel processes; determining sub-training data sets corresponding to the data parallel processes based on the model state quantities matched with the data parallel processes; executing the data parallel processes; determining gradients of the network model to be updated on different data parallel processes according to the sub-training data sets corresponding to the data parallel processes; performing gradient reduction processing on the gradients to obtain corresponding gradient reduction data; and updating model parameters of the network model to be updated according to the gradient reduction data to obtain a trained network model. According to the method, part of the model state quantities is updated and saved for each data parallel process, the transmission efficiency between training nodes can be balanced, and the training efficiency of the network model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model parameter updating method, apparatus, computer device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology and the widespread application of neural network models in various fields, including image recognition, speech recognition, and natural language processing, the structure of neural networks has become increasingly complex as they have further developed. The scale of network parameters and the amount of computation have also increased. It takes a lot of time to train neural networks to obtain an applicable network model. Therefore, it is necessary to speed up model training and reduce model training time and resource consumption.

[0003] Traditionally, training large neural network models with complex structures usually adopts a model parallel approach, which involves splitting the neural network model into multiple components and deploying each component on different training machines for simultaneous training, in order to reduce training time and speed up the training of neural network models.

[0004] However, parallel modeling often suffers from poor performance due to the large size of neural network models, requiring splitting them across multiple training machines. This makes it impossible to consider the training status of all machines simultaneously. Furthermore, the synchronization time between nodes remains a problem in order to ensure synchronous operation across all training machines. Therefore, traditional methods for accelerating neural network model training still suffer from poor parallel training performance and long training times. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for updating model parameters that can reduce the training time of neural network models and improve training efficiency, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for updating model parameters. The method includes:

[0007] Obtain the model state variables of the network model to be updated, and assign each of the model state variables to a separate data parallel process;

[0008] Based on the model state variables that match each of the data parallel processes, a sub-training dataset corresponding to each of the data parallel processes is determined.

[0009] Each of the aforementioned data parallel processes is executed, and the gradient of the network model to be updated on different data parallel processes is determined based on the sub-training datasets that correspond one-to-one with each of the aforementioned data parallel processes.

[0010] Gradient reduction processing is performed on each of the gradients to obtain gradient reduction data that matches different parallel data processes;

[0011] Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain the trained network model.

[0012] In one embodiment, the network model to be updated includes a first linear layer and a second linear layer; the step of structural segmentation based on the network model to be updated to obtain multiple model components includes: vertical segmentation based on the first linear layer to obtain multiple first-type model components corresponding to the first linear layer, and horizontal segmentation based on the second linear layer to obtain multiple second-type model components corresponding to the second linear layer.

[0013] The execution of each of the model parallel processes, based on the model state variables, determines the gradient tensor corresponding to the model parallel process, including:

[0014] Execute the parallel process of each model corresponding to each of the first type of model components, and based on the first activation function corresponding to the first linear layer, perform forward and backward calculations of each of the first type of model components according to the model state variables to obtain the first output data of each of the first type of model components.

[0015] Execute the parallel process of each model corresponding to each of the second type of model components, and obtain the second output data of each of the second type of model components based on the second activation function corresponding to the second linear layer and the forward and backward computation of each of the first output data processes of the second type of model components.

[0016] Each of the second output data is determined as a gradient tensor corresponding to a parallel process of each model.

[0017] In one embodiment, obtaining the order of the training nodes in the pipeline group, such that each training node performs forward and backward computation sequentially according to the order to obtain the average model gradient, includes:

[0018] The full training set corresponding to the network model to be updated is divided into multiple batches of micro-training sets;

[0019] The order of each training node in the pipeline group is obtained; the training node includes a first training node and a second training node, with the first training node preceding the second training node; the first training node is used to obtain micro-training sets of each batch in sequence, perform forward calculation to obtain a loss value, and send the loss value to the second training node; the second training node is used to perform backward calculation based on the loss value to obtain gradient data;

[0020] While the second training node performs reverse calculation based on the loss value, the first training node obtains the next batch of micro-training sets and performs forward calculation in parallel.

[0021] Secondly, this application also provides a model parameter updating device. The device includes:

[0022] The model state quantity acquisition module is used to acquire the model state quantity of the network model to be updated, and to divide each of the model state quantities into different data parallel processes.

[0023] The sub-training dataset determination module is used to determine the sub-training dataset that corresponds one-to-one with each of the data parallel processes based on the model state variables that match each of the data parallel processes.

[0024] The gradient determination module is used to execute each of the data parallel processes and determine the gradient of the network model to be updated on different data parallel processes based on the sub-training datasets that correspond one-to-one with each of the data parallel processes.

[0025] The gradient reduction processing module is used to perform gradient reduction processing on each of the gradients to obtain gradient reduction data that matches different parallel data processes.

[0026] The model parameter update module is used to update the model parameters of the network model to be updated based on the gradient reduction data, so as to obtain the trained network model.

[0027] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0028] Obtain the model state variables of the network model to be updated, and assign each of the model state variables to a separate data parallel process;

[0029] Based on the model state variables that match each of the data parallel processes, a sub-training dataset corresponding to each of the data parallel processes is determined.

[0030] Each of the aforementioned data parallel processes is executed, and the gradient of the network model to be updated on different data parallel processes is determined based on the sub-training datasets that correspond one-to-one with each of the aforementioned data parallel processes.

[0031] Gradient reduction processing is performed on each of the gradients to obtain gradient reduction data that matches different parallel data processes;

[0032] Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain the trained network model.

[0033] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0034] Obtain the model state variables of the network model to be updated, and assign each of the model state variables to a separate data parallel process;

[0035] Based on the model state variables that match each of the data parallel processes, a sub-training dataset corresponding to each of the data parallel processes is determined.

[0036] Each of the aforementioned data parallel processes is executed, and the gradient of the network model to be updated on different data parallel processes is determined based on the sub-training datasets that correspond one-to-one with each of the aforementioned data parallel processes.

[0037] Gradient reduction processing is performed on each of the gradients to obtain gradient reduction data that matches different parallel data processes;

[0038] Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain the trained network model.

[0039] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0040] Obtain the model state variables of the network model to be updated, and assign each of the model state variables to a separate data parallel process;

[0041] Based on the model state variables that match each of the data parallel processes, a sub-training dataset corresponding to each of the data parallel processes is determined.

[0042] Each of the aforementioned data parallel processes is executed, and the gradient of the network model to be updated on different data parallel processes is determined based on the sub-training datasets that correspond one-to-one with each of the aforementioned data parallel processes.

[0043] Gradient reduction processing is performed on each of the gradients to obtain gradient reduction data that matches different parallel data processes;

[0044] Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain the trained network model.

[0045] In the aforementioned model parameter update method, apparatus, computer device, storage medium, and computer program product, the model state variables of the network model to be updated are obtained, and each model state variable is divided into different data parallel processes. Then, based on the model state variables matching each data parallel process, a sub-training dataset corresponding to each data parallel process is determined. Further, by executing each data parallel process, the gradients of the network model to be updated on different data parallel processes are determined according to the sub-training datasets corresponding to each data parallel process. Gradient reduction processing is then performed on each gradient to obtain gradient reduction data matching different data parallel processes. Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain a trained network model. This achieves the determination of sub-training datasets matching each data parallel process based on the model state variables of the network model to be updated, and updates the model parameters only for the matching sub-training datasets, without needing to save and update the complete model state variables for each data process. This effectively balances the transmission efficiency between training nodes on the training machine, saves the GPU memory usage of the training machine, reduces the training time of the neural network model, and improves the training efficiency of the network model. Attached Figure Description

[0046] Figure 1 This is a diagram illustrating the application environment of a model parameter update method in one embodiment.

[0047] Figure 2 This is a flowchart illustrating a model parameter update method in one embodiment;

[0048] Figure 3 This is a schematic diagram of parallel data communication processing in one embodiment;

[0049] Figure 4 This is a schematic diagram of state variable segmentation corresponding to a data parallel process in one embodiment;

[0050] Figure 5 This is a schematic diagram of gradient reduction processing based on a data-parallel process in one embodiment;

[0051] Figure 6 This is a schematic diagram of full aggregation processing based on data parallel processes in one embodiment;

[0052] Figure 7 This is a flowchart illustrating the model parameter update method in another embodiment;

[0053] Figure 8 This is a schematic diagram of parallel communication processing in one embodiment;

[0054] Figure 9 This is a flowchart illustrating the model parameter update method in another embodiment;

[0055] Figure 10 This is a schematic diagram of pipelined parallel communication processing in one embodiment;

[0056] Figure 11 This is a structural block diagram of a model parameter update device in one embodiment;

[0057] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] The model parameter update method provided in this application involves artificial intelligence (AI) technology. AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. As a comprehensive discipline, AI technology covers a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0060] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0061] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0062] The model parameter update method provided in this application specifically relates to deep learning technology in artificial intelligence and can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, aircraft, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0063] Furthermore, both terminal 102 and server 104 can be used independently to execute the model parameter update method provided in this embodiment, or they can work together to execute the model parameter update method provided in this embodiment. For example, taking the collaborative execution of the model parameter update method provided in this embodiment by terminal 102 and server 104 as an example, server 104 obtains the model state variables of the network model to be updated and divides each model state variable into different data parallel processes. Then, based on the model state variables matching each data parallel process, it determines the sub-training dataset corresponding to each data parallel process. Further, server 104 executes each data parallel process, determines the gradients of the network model to be updated on different data parallel processes based on the sub-training datasets corresponding to each data parallel process, and performs gradient reduction processing on each gradient to obtain gradient reduction data matching different data parallel processes. Finally, server 104 can update the model parameters of the network model to be updated based on the gradient reduction data to obtain a trained network model. After obtaining the trained network model, it can be further deployed to terminal 102 for application. The model state of the network model to be updated, as well as the sub-training dataset, can be stored in the local storage of the terminal 102, the data storage system, or the cloud storage of the server 104.

[0064] In one embodiment, such as Figure 2 As shown, a model parameter update method is provided. The method is illustrated using an example where it is executed by a computer device. It can be understood that the computer device can be... Figure 1 The terminal 102 shown can also be a server 104 with a gateway plugin 106 deployed, or a system consisting of terminal 102 and server 104, and is implemented through interaction between terminal 102 and server 104. In this embodiment, the model parameter update method includes:

[0065] Step S202: Obtain the model state variables of the network model to be updated, and assign each model state variable to a separate data parallel process.

[0066] The model state variables of the network model to be updated correspond to the specific network model. For example, if the complete network model to be updated is represented by y = ax + b, then "a" and "b" can be understood as the model state variables (i.e., model states). Specifically, the model state variables are used to store the values ​​submitted to the server and the set of validation errors associated with the submitted values. Specifically, they can include optimizer state variables and gradient data during model training. Depending on the specific network model and the training sample set, the optimizer used and the gradient data obtained during training will be different.

[0067] Similarly, the state variables "a" and "b" correspond to multiple tensors. A tensor is a multidimensional matrix and is the basic unit of computation running on a GPU (Graphics Processing Unit). Since network models usually have multiple layers, including stateless layers and stateful layers, the stateful layers are specifically represented by the weights of each layer. The weights are one or more tensors learned using stochastic gradient descent, which contain the content or knowledge in the network model.

[0068] Furthermore, during network model training, the optimizers used often employ stochastic gradient descent, which is mainly used to calculate the gradient of the loss with respect to the network parameters (where the gradient can be represented as the derivative of tensor operations). The weight parameters are then moved in the opposite direction of the gradient. The purpose of applying the gradient descent algorithm is also to reduce data loss and improve the model training accuracy.

[0069] For example, if the network model to be updated is a natural language processing model (i.e., an NLP model), and the optimizer used is the Adam optimizer, the Adam optimizer, or Adaptive Moment Estimation, is essentially an RMSprop algorithm with a momentum term (i.e., root mean square propagation, an adaptive learning rate method used to optimize the loss function to avoid excessive fluctuations during updates and further accelerate the convergence speed of the function). It can be understood as an optimizer that absorbs the advantages of adaptive learning rate gradient descent and momentum gradient descent algorithms, which can adapt to sparse gradients (i.e., natural language and computer vision problems) and alleviate the gradient oscillation problem.

[0070] In the model training process, the optimizer updates the network parameters based on the gradient information obtained from backpropagation, thereby reducing the calculated value of the loss function. The optimizer's state variables can be understood as the state information of the optimizer during the management and updating of the model, specifically including cached and used parameters, and managed parameter sets. Cached and used parameters may include, for example, the gradients used in each update operation.

[0071] Specifically, in this embodiment of the application, the training node on the training machine (i.e. the GPU card deployed on the training machine) obtains the model state variables of the network model to be updated, as well as the tensors corresponding to each model state variable, and counts the total number of tensors corresponding to each model state variable. Then, based on the total number of tensors obtained, the tensors corresponding to each model state variable are divided equally and assigned to each data parallel process.

[0072] In one embodiment, such as Figure 3 As shown, a processing flow for parallel data communication is provided, referring to... Figure 3 It can be seen that the processing flow of parallel data communication includes: Figure 3 The data shown in Figure (a) is used for parallel model state variable partitioning. Figure 3 The data parallel process gradient reduction shown in Figure (b) and... Figure 3 The data parallel process weight aggregation is shown in Figure (c).

[0073] Specifically, in the data parallel communication process, unlike the traditional allreduce communication method, where each data parallel process does not need to save and update a complete copy of the model state, a further memory-saving approach is adopted. This mainly optimizes the memory usage of Model States by dividing the network model's model state among multiple different data parallel processes. This changes the approach from "each data parallel process saving a complete copy of the model state" to "each data parallel process maintaining only a portion of the model state", thus saving memory.

[0074] Among them, reference Figure 3 As shown in Figure (a), during the parallel data communication process, the model state variables of the network model are divided among all data parallel processes. Usually, an equal division method is adopted, that is, by obtaining the tensor corresponding to each model state variable and counting the total number of tensors corresponding to each model state variable, the tensor corresponding to each model state variable is divided equally according to the total number of tensors obtained, and the tensor corresponding to each model state variable is assigned to each data parallel process.

[0075] Similarly, refer to Figure 3 As shown in Figure (b), for each data parallel process, a corresponding sub-training dataset is run. By executing each data parallel process, the network model on that data parallel process is updated in parallel based on the sub-training dataset matched to that data parallel process, and gradient reduction processing is performed. Specifically, ReduceScatter processing (i.e., scattering reduction processing) is used for each data parallel process to obtain the reduced gradient corresponding to each data parallel process, that is, to obtain the partial gradient reduction data corresponding to each data parallel process.

[0076] Furthermore, referring to Figure 3 As shown in Figure (c), for each data parallel process, the corresponding partial gradient reduction data is processed, and then allgather processing (i.e., full aggregation processing) is required so that each data parallel process can obtain the full weight information, thereby achieving parameter update of the network model based on the full weight information and obtaining a trained network model.

[0077] Step S204: Based on the model state variables that match each data parallel process, determine the sub-training dataset that corresponds one-to-one with each data parallel process.

[0078] Specifically, by equally dividing the multiple tensors corresponding to each model state variable into each data parallel process, the tensor matching each data parallel process is determined, and then the sub-training dataset corresponding to each data parallel process is further determined based on the tensor.

[0079] In this setup, multiple data-parallel processes correspond to a complete training set, and the complete training set can be divided into multiple smaller training sets, i.e., multiple sub-training sets, each corresponding to a data-parallel process. Since data parallel processing is required, one data-parallel process can run on each GPU card; that is, there is a one-to-one correspondence between GPU cards and data-parallel processes.

[0080] For example, if the complete network model to be updated is represented by y = ax + b, then multiple GPUs are needed to train the network model y = ax + b simultaneously. Furthermore, the GPUs need to interact with each other, meaning that the training results of each sub-training set used by each data parallel process need to be exchanged to obtain the complete training results of the network model to be updated.

[0081] In one embodiment, such as Figure 4 As shown, a schematic diagram of state variable partitioning corresponding to a data parallel process is provided, referencing... Figure 4 As we can see, if a training machine has 64 GPUs, then each GPU corresponds to a data parallel process rank. For example, if there are N tensors, GPU 0 corresponds to rank 0. When partitioning the state variables for the data parallel process, rank 0 is partitioned into tensor 1, and so on. GPU 1 corresponds to rank 1, and rank 1 is partitioned into tensor 2, tensor 3, and so on, until all data parallel processes have been partitioned into various tensors.

[0082] In the process of data parallel processing, each tensor is only "visible" to a certain data parallel process. That is, each data parallel process has its own specific parts that need to be saved and updated. For example, if rank 0 updates tensor1, then tensor1 is only visible to rank 0. And so on. Each GPU card only needs to update the tensors that are visible to itself, and it also only needs to save the training results that are visible to itself.

[0083] Step S206: Execute each data parallel process and determine the gradient of the network model to be updated on different data parallel processes based on the sub-training datasets that correspond one-to-one with each data parallel process.

[0084] Specifically, by executing each data parallel process, the forward computation of the network model to be updated is performed based on the sub-training dataset corresponding to each data parallel process to obtain the training loss value. Furthermore, based on the training loss value, the backward computation of the network model to be updated is performed to obtain the gradient of the network model to be updated on different data parallel processes.

[0085] The training process of the network model specifically includes forward computation and backward computation. Forward computation is used to obtain the training loss value, and then the training loss value is used for backward computation to obtain the gradient during the model training process. The optimizer is then used to update the weights corresponding to each gradient. In other words, the optimizer can update the network parameters based on the gradient information propagated from the backpropagation of the network model, and at the same time, it plays a role in reducing the calculated value of the loss function during the training process.

[0086] Step S208: Perform gradient reduction processing on each gradient to obtain gradient reduction data that matches different parallel data processes.

[0087] Specifically, the ReduceScatter process (i.e., scattering reduction process) is used for each data parallel process. That is, by reducing the gradient of each data process, the reduced gradient corresponding to each data parallel process is obtained, that is, the partial gradient reduction data corresponding to each data parallel process is obtained.

[0088] Furthermore, gradients are generated during the reverse computation process, and gradients can be represented as the derivatives of tensor operations. Consequently, each calculated gradient corresponds to a gradient tensor. During data parallel processing, since each tensor is only "visible" to a specific data parallel process (meaning each data parallel process has its own specific parts that need to be saved and updated), the data parallel process determines which data parallel process the gradient tensor is "visible" to based on the tensors visible to it. Then, it calls the reduce-scatter interface (i.e., the scattering reduction processing interface) through the NVIDIA NCCL GPU communication library (i.e., NVIDIA Collective Communication Library) to perform gradient reduction operations.

[0089] The reduce-scatter interface is used for scatter reduction processing. Scatter reduction means that each processing node divides its data into equally sized blocks, and each node performs a reduction operation based on the data obtained from the index (i.e., a function that returns values ​​or references to values ​​in a table or range). In other words, first, scattering (Scatter) is performed, dividing the data in the server into equally sized blocks, and then reducing (Reduce) the parameter data obtained by each server according to the sequence (Rank Index). This reduction processing can be summation or finding the maximum value, etc.

[0090] In one embodiment, such as Figure 5 As shown, a schematic diagram of gradient reduction processing based on data parallel processes is provided. (Refer to...) Figure 5As can be seen, for example, if there are 4 parallel data processes rank 0 to rank 3, and the index functions corresponding to the 4 parallel data processes are in 0 to in 3 respectively, by performing scattering reduction on the 4 parallel data processes, the output results corresponding to the 4 parallel data processes can be obtained as out 0 to out 3. Then, after the back-play is completed, by performing gradient reduction, each parallel data process can save a part of the reduced gradient data.

[0091] Among them, reference Figure 5 It can be seen that, for each data parallel process, a certain reduction operation (such as summation) is performed on the data block to be processed, and the corresponding reduction result is obtained. The result is then stored in the memory of the processing node (i.e., GPU card) corresponding to each data parallel process.

[0092] Step S210: Update the model parameters of the network model to be updated based on the gradient reduction data to obtain the trained network model.

[0093] Specifically, based on the gradient reduction data, weights corresponding to each parallel process are determined. Then, based on the weights corresponding to each parallel process, full aggregation is performed to obtain full weight information. Based on the full weight information, the parameters of the network model to be updated are updated to obtain the trained network model.

[0094] Since each data-parallel process saves a portion of the reduced gradient data after backpropagation, each process can further call the Adam optimizer to update the weights corresponding to these gradients. The optimizer updates the network parameters based on the gradient information from backpropagation, thereby reducing the calculated value of the loss function.

[0095] Furthermore, based on the gradient reduction data, the steps for determining the weights corresponding to each parallel process of the data are as follows:

[0096] Based on the gradient reduction data corresponding to each data parallel process, the first-order momentum data and second-order momentum data corresponding to each data parallel process are determined respectively; based on the first-order momentum data and the first-order momentum decay coefficient, the first bias correction data of the first-order momentum data is determined, and based on the second-order momentum data and the second-order momentum decay coefficient, the second bias correction data of the second-order momentum data is determined; based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, the weights corresponding to each data parallel process are determined respectively.

[0097] Specifically, this involves updating the network model's weights (i.e., updating the network model's parameters) using the Adam optimizer. The gradient update rule of the Adam optimizer is reflected in its update formula. In essence, the Adam optimizer can dynamically adjust the learning rate of each parameter using the first and second momentum data of the gradient, achieving the goal of updating model parameters according to different learning rates.

[0098] Specifically, the first-order momentum data and second-order momentum data corresponding to each data parallel process are determined using the following formulas (1) and (2):

[0099] m t =β1m t-1 +(1-β1)Δw; (1)

[0100] v t =β2v t-1 +(1-β2)Δw 2 (2)

[0101] Where t is the update step size, m t To update the first-order momentum data at step size t, β1 is the first-order momentum decay coefficient, m t-1 To update the first-order momentum data at step size t-1, Δw is the gradient reduction data, v t To update the second-order momentum data at step size t, β2 is the second-order momentum decay coefficient, v t-1 To update the second-order momentum data at a step size of t-1.

[0102] Furthermore, specifically using the following formula (3), the first bias correction data of the first-order momentum data is determined based on the first-order momentum data and the first-order momentum decay coefficient. Based on the following formula (4), the second bias correction data of the second-order momentum data is determined based on the second-order momentum data and the second-order momentum decay coefficient.

[0103]

[0104]

[0105] Where t is the update step size. To update the first bias correction data with a step size of t, m t To update the first-order momentum data at step size t, β1 is the first-order momentum decay coefficient. To update the second bias correction data with a step size of t, v t To update the second-order momentum data at step size t, β2 is the second-order momentum decay coefficient.

[0106] Similarly, using the following formula (5), based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, the weights corresponding to each parallel process of the data are determined respectively:

[0107]

[0108] Where t is the update step size, W t To update the weights at step size t, W t-1 To update the weights at step size t-1, α is the learning rate. To update the first bias correction data with a step size of t, To update the second bias correction data with a step size of t, ε is a hyperparameter in the weight update process, a constant added to maintain numerical stability.

[0109] In one embodiment, after determining the weights corresponding to each data parallel process based on the gradient reduction data, each data parallel process performs full aggregation processing by calling the allgather interface of the NVIDIA NCCL GPU communication library, based on the weights corresponding to each data parallel process, to obtain full weight information.

[0110] In this context, after each processing node (i.e., GPU card) performs scattering, each processing node obtains a data block from other processing nodes. The operation of piecing together the data blocks obtained by a processing node is called gathering. The operation in which all processing nodes piece together the data blocks they receive (i.e., all perform the gathering operation) is called allgather.

[0111] Specifically, such as Figure 6 As shown, a processing procedure for full aggregation based on data parallel processes is provided, referring to... Figure 6 As can be seen, for example, if there are 4 data parallel processes rank 0 to rank 3, and the input corresponding to each data parallel process is in 0 to in 3, then by performing full aggregation processing on the weights corresponding to each data parallel process, each data parallel process can obtain all the data blocks (including the complete data blocks obtained by splicing in 0 to in 3). After performing full aggregation processing, each data parallel process can obtain the weight information of all other data parallel processes, that is, obtain the full weight information.

[0112] Furthermore, after obtaining the full weight information, the parameters of the network model to be updated are updated based on the full weight information to obtain a trained network model.

[0113] In the aforementioned model parameter update method, the model state variables of the network model to be updated are obtained, and each model state variable is assigned to a separate data parallel process. Then, based on the model state variables matching each data parallel process, a sub-training dataset corresponding to each data parallel process is determined. Further, by executing each data parallel process, the gradients of the network model to be updated on different data parallel processes are determined based on the sub-training datasets corresponding to each data parallel process. Gradient reduction processing is then performed on each gradient to obtain gradient reduction data matching different data parallel processes. Based on this gradient reduction data, the model parameters of the network model to be updated are updated, resulting in a trained network model. This method achieves the determination of sub-training datasets matching each data parallel process based on the model state variables of the network model to be updated, and updates the model parameters only for the matching sub-training datasets, without needing to save and update the complete model state variables for each data process. This effectively balances the transmission efficiency between training nodes on the training machine, saves GPU memory usage, reduces the training time of the neural network model, and improves the training efficiency of the network model.

[0114] In one embodiment, such as Figure 7 As shown, a method for updating model parameters is provided, which specifically includes the following steps:

[0115] Step S702: Perform structural segmentation based on the network model to be updated to obtain multiple model components.

[0116] Specifically, the network model to be updated includes a first linear layer and a second linear layer. When performing structural segmentation based on the network model to be updated, the segmentation is performed vertically based on the first linear layer to obtain multiple first-type model components corresponding to the first linear layer, and horizontally based on the second linear layer to obtain multiple second-type model components corresponding to the second linear layer.

[0117] In one embodiment, taking a transformer model as an example, the transformer model integrates an MLP (Multilayer Perceptron) network and a Self-Attention network. The MLP network includes a first linear layer and a second linear layer; therefore, structural segmentation needs to be performed on the first and second linear layers of the MLP network. Similarly, the Self-Attention network also includes a first linear layer and a second linear layer; therefore, structural segmentation also needs to be performed on the first and second linear layers of the Self-Attention network.

[0118] Step S704: Divide each model component into a model parallel process, with each model component and model parallel process corresponding one-to-one.

[0119] Specifically, for each model parallel process, a model component is divided to correspond to it one by one. The model component includes multiple first-type model components corresponding to the first linear layer and multiple second-type model components corresponding to the second linear layer. Thus, the model parallel process also includes the model parallel process corresponding to the first-type model component and the model parallel process corresponding to the second-type model component.

[0120] Step S706: Execute the parallel processes of each model and determine the gradient tensor corresponding to the parallel process of the model based on the model state variables.

[0121] Specifically, by executing the parallel processes of each model corresponding to each first type of model component, and based on the first activation function corresponding to the first linear layer, the forward and backward computations of each first type of model component are performed according to the model state variables to obtain the first output data of each first type of model component.

[0122] Similarly, by executing the parallel processes of each model corresponding to each second type of model component, based on the second activation function corresponding to the second linear layer, and according to the forward and backward computation of each first output data process of the second type of model component, the second output data of each second type of model component is obtained, and then each second output data is determined as a gradient tensor corresponding to each parallel process of the model.

[0123] In one embodiment, such as Figure 8 As shown, a schematic diagram of parallel communication processing in a model is provided, with reference to... Figure 8 As can be seen, this embodiment takes a transformer model that integrates an MLP network and a Self-Attention network as an example, combined with... Figure 8 Figure (a) in the figure provides a detailed explanation of the structural segmentation and gradient tensor determination for MLP networks:

[0124] Model parallelism is achieved by partitioning the Weights Tensor (i.e., the model parameters and structure) of the network model. Specifically, when partitioning the structure of the MLP network in the transformer network, equivalent matrix factorization is used for equivalent expression to minimize communication. In this embodiment, the additional communication for the model parallel process is represented by functions f and g, where function f is the identity function (i.e., the identity mapping function) and function g is the all-gather function (i.e., the full-gather function).

[0125] For example, if an MLP network is represented by Y = cx + d, then c and d are both state variables of the network model. Assuming c corresponds to the first linear layer and d corresponds to the second linear layer, when structurally partitioning c and d, vertical partitioning is performed based on the first linear layer to obtain multiple first-type model components corresponding to the first linear layer, and horizontal partitioning is performed based on the second linear layer to obtain multiple second-type model components corresponding to the second linear layer. For instance, if four GPUs are used, after partitioning the network model structure, each GPU will not store the full model; each GPU will store a portion of the network model structure or a portion of the network model parameters.

[0126] Specifically, when executing the parallel process of each model corresponding to each first type of model component, based on the first activation function corresponding to the first linear layer, and according to the model state variables, the forward and backward computations of each first type of model component are performed by using function f to perform forward computation (i.e., forward) and backward computation (i.e., backward) on the first linear layer.

[0127] In the forward computation, the function f, i.e., the identity mapping function, is used, which can be understood as distributing the same input data X (i.e., copying X to different GPU cards) to different GPU cards. In the backward computation, the function g, i.e., the full aggregation function, is used. The gradients of each GPU card with respect to the input data X need to be summed to obtain the new gradients with respect to the input data X, which are the first output data of each first-type model component.

[0128] Furthermore, referring to Figure 8 As shown in Figure (a), the activation function of the first linear layer is a non-linear activation function, specifically represented by Y = GeLU. The input data X is copied using an identity mapping function, and the same input data X is input into different parallel model processes. However, due to the structural segmentation of the network model, the model structure and parameters are not consistent across each parallel model process, requiring a combination of... Figure 8 Considering the model components A1 and A2 shown in Figure (a), since the model parameters are inconsistent in the parallel process of different models, that is, A1 and A2 are not consistent, and thus XA1 and XA2 are also different, resulting in different gradients being returned. Therefore, the gradients need to be added together to obtain the updated gradient with respect to the input data X.

[0129] Specifically, the all-reduce function (i.e., the full reduction function, including scattering reduction and weight aggregation) is used for update processing to obtain the first output data of each first-type model component. The output results can include... Figure 8 Y1 and Y2 in Figure (a).

[0130] Similarly, parallel processes corresponding to each second-type model component are executed. Based on the second activation function corresponding to the second linear layer, and according to the forward and backward computations of each first output data process for the second-type model component, the second output data of each second-type model component is obtained. Specifically, the function g is used to perform forward computation and backward computation on the second linear layer. Then, each second output data is determined as a gradient tensor corresponding one-to-one with each parallel process of the model. The activation function of the second linear layer is a non-linear function, specifically represented by Z = Dropout(YB).

[0131] In the forward computation, the function g, i.e., the full aggregation function, is applied. Specifically, for the first output data Y1 and Y2 of the first linear layer, it is necessary to consider the model components B1 and B2 obtained by horizontally partitioning the second linear layer. By combining Y1B1 and Y2B2, and further calling the function g, the computation is performed on the model components Y1B1 and Y2B2. Figure 8 When Y1B1, Z1 and Y2B2, Z2 in Figure (a) are aggregated, they are aggregated according to the last dimension Z to obtain the complete Z. The complete Z can be understood as the result of the forward calculation, i.e. the loss value.

[0132] Furthermore, backward computation can be understood as a simple cut, that is, a direct copy and redistribution method, similar to the identity processing method, meaning that the second output data of each second type of model component obtained on each GPU card is the same. Then, each second output data is determined as a gradient tensor corresponding one-to-one with each parallel process of the model, meaning that each GPU card will obtain the same copied gradient tensor.

[0133] In one embodiment, combined Figure 8 Figure (b) in the diagram provides a detailed explanation of the structural segmentation and gradient tensor determination for the Self-Attention network:

[0134] In this embodiment, when performing structural segmentation on the Self-Attention network, equivalent matrix factorization is used for equivalent representation to minimize communication. Additional communication for the parallel model process is represented by functions f and g, where function f is the identity function (i.e., the identity mapping function), and function g is the allgather function (i.e., the allgather function).

[0135] Specifically, when performing structural segmentation on the first and second linear layers of the Self-Attention network, the first linear layer is segmented vertically to obtain multiple first-class model components corresponding to it, and the second linear layer is segmented horizontally to obtain multiple second-class model components corresponding to it. Since the Self-Attention network is a multi-head attention mechanism, each attention head can be further segmented, referring to... Figure 8 As shown in Figure (b), three linear layers related to each attention head can be obtained, which can be converted into Q, K, and V. When dividing Q, K, and V, we can obtain [Q1, Q2], [K1, K2], and [V1, V2].

[0136] Furthermore, for the Self-Attention network, the parallel processes of each model corresponding to each first-type model component are executed. Based on the first activation function corresponding to the first linear layer, and according to the model state variables, when performing forward and backward computations of each first-type model component, specifically, function f is used for the first linear layer (i.e., Figure 8 In Figure (b), forward computation is performed on the Q, K, and V layers, and backward computation is performed on the first linear layer (Q, K, and V layers).

[0137] During the forward computation, the function f, i.e. the identity mapping function, is applied, which can be understood as distributing the same input data X (i.e., copying X to different GPU cards) to different GPU cards.

[0138] Specifically, the same input data X is first copied multiple times and then distributed to multiple GPU cards. In the multi-head attention mechanism, when the three linear layers of each attention head are converted into Q, K, and V layers, a linear layer of h to 3h can be used, and the layers are divided into three equal parts according to the last dimension to obtain the Q, K, and V layers.

[0139] Furthermore, calculations are performed separately for the Q, K, and V layers. as well as Then as well as The specific values ​​are input into the softmax layer, and then into the dropout layer to obtain the result. as well as The corresponding attention scores are then multiplied by V1, and another attention score is multiplied by V2 to obtain the first output data of the parallel processes of different models, namely Y1 and Y2. The activation function of the first linear layer of the Self-Attention network is represented by the line Y = Self-Attention(X).

[0140] Similarly, the computational processing for the second linear layer of the Self-Attention network is similar to that for the second linear layer of the MLP network. Specifically, a function g is used to perform forward computation and backward computation on the second linear layer. Then, each second output data is determined as a gradient tensor corresponding one-to-one with each parallel process of the model. The activation function of the second linear layer is a non-linear function, specifically represented by Z = Dropout(YB).

[0141] In this process, the second output data of each type of model component obtained on each GPU card is the same. Specifically, each second output data is determined as a gradient tensor that corresponds one-to-one with each parallel process of the model. That is, each GPU card will obtain the same gradient tensor obtained by copying.

[0142] Step S708: Update the model parameters of the network model to be updated based on the gradient tensor to obtain the trained network model.

[0143] Specifically, since the determined gradient tensor has undergone full aggregation processing, meaning that each parallel process of the model can obtain the gradient tensors of other parallel processes of the model, the determined gradient tensor can be understood as the full gradient tensor. Therefore, based on the full gradient tensor, the model structure and model parameters of the model to be updated can be updated to obtain the trained network model.

[0144] In this embodiment, structural segmentation based on the network model to be updated yields multiple model components. Each model component is then assigned to a parallel model process, and these parallel processes are executed. Based on the model state variables, the gradient tensor corresponding to each parallel process is determined. Subsequently, the model parameters of the network model to be updated are updated according to the gradient tensor, resulting in a trained network model. This approach achieves the determination of multiple model components based on the network model's structure, and sets up parallel training processes for each component to quickly determine the gradient tensor corresponding to the parallel process. The model parameters of the network model to be updated are then updated based on the gradient tensor. In other words, model parallelism reduces the time required for model training and improves the efficiency of model parameter updates.

[0145] In one embodiment, such as Figure 9This paper provides a method for updating model parameters, which includes the following steps:

[0146] Step S902: The training nodes corresponding to each data parallel process are divided into multiple pipeline groups. Each pipeline group includes at least two training nodes, and the number of training nodes in each pipeline group is the same.

[0147] Specifically, by obtaining training nodes that correspond one-to-one with each data parallel process, and dividing the training nodes into multiple pipeline groups, for example, setting up two pipeline groups, and setting up two training nodes in each pipeline group.

[0148] Step S904: Obtain the sorting of each training node in the pipeline group, so that each training node performs forward and backward calculations in sequence according to the sorting to obtain the average model gradient.

[0149] Specifically, the full training set corresponding to the network model to be updated is divided into multiple batches of micro-training sets, and the sorting of each training node in the pipeline group is obtained.

[0150] Furthermore, the training nodes include a first training node and a second training node, with the first training node preceding the second training node. The first training node is used to sequentially acquire each batch of micro-training sets, perform forward computation to obtain the loss value, and send the loss value to the second training node. The second training node is used to perform backward computation based on the loss value to obtain gradient data.

[0151] During the reverse calculation based on the loss value at the second training node, the first training node obtains the next batch of micro-training sets and performs forward calculations in parallel.

[0152] In one embodiment, such as Figure 10 As shown, a schematic diagram of pipelined parallel communication processing is provided, with reference to... Figure 10 It can be seen that there are two pipeline groups, each pipeline group includes two training nodes, namely GPU 0 and GPU2 are in one pipeline group, and GPU 1 and GPU3 are in the other pipeline group.

[0153] Furthermore, GPU 0 reads a minibatch of data (i.e., micro-batch training data), performs forward computation of F0, obtains activation data (i.e., intermediate result data, which can be understood as the loss function value obtained from forward computation, used for backward computation to obtain the gradient), and sends the calculated activation data to GPU 2. GPU 2 then performs backward computation of B0 based on the activation data to obtain the gradient data.

[0154] During the B0 backward computation on GPU 2, GPU 0 simultaneously reads the next minibatch data and performs the F1 forward computation. This parallelism between the F1 forward computation and the B0 backward computation, which have no data flow dependency, achieves pipeline parallelism. Each minibatch is obtained by splitting the complete dataset. The batch size (number of samples in a batch) determines the size of the resulting minibatch. For example, if the complete dataset has 64 samples to be processed, and the pipeline has two stages and two training nodes, then 32 samples are first read as a minibatch and fed to GPU 0 for the F0 forward computation. While GPU 2 is performing the B0 backward computation, a second minibatch of 32 samples is obtained and fed to GPU 0 for the F1 forward computation.

[0155] Similarly, the computational processing operations performed on another pipeline group, including the pipeline group with two training nodes, GPU 1 and GPU 3, are the same as those performed on the pipeline group including GPU 0 and GPU 2. Within the same pipeline group, activation (forward computation, sending intermediate data) or gradients (backward computation, obtaining gradient data) is sent via broadcast. Furthermore, only the first stage of the pipeline requires reading training data (i.e., minibatch data), while the last stage requires reading label data (i.e., the original labeled data of the training samples) to calculate the loss value.

[0156] in, Figure 10 The forward computation shown here includes F0 to F3, and the backward computation includes B0 to B3. This is only used to represent a specific case of the number of forward and backward computations during model training. In actual applications, the number of forward and backward computations is not limited to specific values ​​and can be set according to actual needs.

[0157] Step S906: Based on the average model gradient, update the model parameters of the network model to be updated to obtain the trained network model.

[0158] Specifically, for the gradient data obtained from different pipeline groupings, it is necessary to further calculate the leather and model gradients, so as to use the average model gradient as the basis for updating the model parameters of the network model to be updated, thereby updating the parameters of the network model and obtaining a trained network model.

[0159] In this embodiment, training nodes corresponding to each data parallel process are divided into multiple pipeline groups, and the order of training nodes in each pipeline group is obtained. Each training node then performs forward and backward computations sequentially according to the order to obtain the average model gradient. Based on this average model gradient, the model parameters of the network model to be updated are then updated, resulting in a trained network model. This achieves parallel forward and backward computations without data flow dependencies between different training nodes based on pipeline groups. This reduces model training time and avoids mutual influence or interference between different training nodes, further improving the accuracy and efficiency of updating model parameters.

[0160] In one embodiment, a model parameter update method is provided, which specifically includes the following steps:

[0161] Obtain the model state variables of the network model to be updated, and assign each model state variable to a separate data parallel process.

[0162] Based on the model state variables that match each data parallel process, a sub-training dataset corresponding to each data parallel process is determined.

[0163] Each data parallel process is executed, and the forward computation of the network model to be updated is performed based on the sub-training dataset that corresponds to each data parallel process to obtain the training loss value.

[0164] Based on the training loss value, the reverse calculation of the network model to be updated is performed to obtain the gradient of the network model to be updated in different data parallel processes.

[0165] Gradient reduction is performed on each gradient to obtain gradient reduction data that matches different parallel data processes.

[0166] Based on the gradient reduction data corresponding to each data parallel process, the first-order momentum data and second-order momentum data corresponding to each data parallel process are determined respectively.

[0167] Based on the first-order momentum data and the first-order momentum decay coefficient, the first bias correction data of the first-order momentum data is determined, and based on the second-order momentum data and the second-order momentum decay coefficient, the second bias correction data of the second-order momentum data is determined.

[0168] Based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, the weights corresponding to the parallel processes of each data are determined.

[0169] Based on the weights that correspond one-to-one with each parallel data process, full aggregation processing is performed to obtain full weight information.

[0170] Based on the full weight information, the parameters of the network model to be updated are updated to obtain the trained network model.

[0171] In the aforementioned model parameter update method, the model state variables of the network model to be updated are obtained, and each model state variable is assigned to a separate data parallel process. Then, based on the model state variables matching each data parallel process, a sub-training dataset corresponding to each data parallel process is determined. Further, by executing each data parallel process, the gradients of the network model to be updated on different data parallel processes are determined based on the sub-training datasets corresponding to each data parallel process. Gradient reduction processing is then performed on each gradient to obtain gradient reduction data matching different data parallel processes. Based on this gradient reduction data, the model parameters of the network model to be updated are updated, resulting in a trained network model. This method achieves the determination of sub-training datasets matching each data parallel process based on the model state variables of the network model to be updated, and updates the model parameters only for the matching sub-training datasets, without needing to save and update the complete model state variables for each data process. This effectively balances the transmission efficiency between training nodes on the training machine, saves GPU memory usage, reduces the training time of the neural network model, and improves the training efficiency of the network model.

[0172] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0173] Based on the same inventive concept, this application also provides a model parameter updating device for implementing the model parameter updating method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more model parameter updating device embodiments provided below can be found in the limitations of the model parameter updating method described above, and will not be repeated here.

[0174] In one embodiment, such as Figure 11As shown, a model parameter update device is provided, including: a model state acquisition module 1102, a sub-training dataset determination module 1104, a gradient determination module 1106, a gradient reduction processing module 1108, and a model parameter update module 1110, wherein:

[0175] The model state quantity acquisition module 1102 is used to acquire the model state quantity of the network model to be updated and to divide each model state quantity into each data parallel process.

[0176] The sub-training dataset determination module 1104 is used to determine the sub-training dataset that corresponds one-to-one with each data parallel process based on the model state variables that match each data parallel process.

[0177] The gradient determination module 1106 is used to execute each data parallel process and determine the gradient of the network model to be updated on different data parallel processes based on the sub-training datasets that correspond one-to-one with each data parallel process.

[0178] The gradient reduction processing module 1108 is used to perform gradient reduction processing on each gradient to obtain gradient reduction data that matches different parallel data processes.

[0179] The model parameter update module 1110 is used to update the model parameters of the network model to be updated based on the gradient reduction data, so as to obtain the trained network model.

[0180] In the aforementioned model parameter update device, the model state variables of the network model to be updated are obtained, and each model state variable is assigned to a separate data parallel process. Then, based on the model state variables matching each data parallel process, a sub-training dataset corresponding to each data parallel process is determined. Further, by executing each data parallel process, the gradients of the network model to be updated on different data parallel processes are determined according to the sub-training datasets corresponding to each data parallel process. Gradient reduction processing is then performed on each gradient to obtain gradient reduction data matching different data parallel processes. Based on this gradient reduction data, the model parameters of the network model to be updated are updated, resulting in a trained network model. This achieves the goal of determining the sub-training datasets matching each data parallel process based on the model state variables of the network model to be updated, and updating the model parameters only for the matching sub-training datasets, without needing to save and update the complete model state variables for each data process. This effectively balances the transmission efficiency between training nodes on the training machine, saves the GPU memory usage of the training machine, reduces the training time of the neural network model, and improves the training efficiency of the network model.

[0181] In one embodiment, the model parameter update module is further configured to: determine the weights corresponding one-to-one with each parallel process of each data according to each gradient reduction data; perform full aggregation processing based on the weights corresponding one-to-one with each parallel process of each data to obtain full weight information; and update the parameters of the network model to be updated according to the full weight information to obtain the trained network model.

[0182] In one embodiment, the gradient determination module is further configured to: execute each data parallel process, perform forward computation of the network model to be updated based on the sub-training dataset corresponding to each data parallel process to obtain the training loss value; and perform backward computation of the network model to be updated based on the training loss value to obtain the gradient of the network model to be updated in different data parallel processes.

[0183] In one embodiment, the model parameter update module is further configured to: determine the first-order momentum data and second-order momentum data corresponding to each data parallel process based on the gradient reduction data corresponding to each data parallel process; determine the first bias correction data of the first-order momentum data based on the first-order momentum data and the first-order momentum decay coefficient, and determine the second bias correction data of the second-order momentum data based on the second-order momentum data and the second-order momentum decay coefficient; and determine the weights corresponding one-to-one with each data parallel process based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated.

[0184] In one embodiment, a model parameter updating apparatus is provided, further comprising:

[0185] The structure partitioning module is used to perform structural segmentation based on the network model to be updated, and obtain multiple model components.

[0186] The component partitioning module is used to divide each model component into a model parallel process; there is a one-to-one correspondence between model components and model parallel processes.

[0187] The gradient tensor determination module is used to execute the parallel processes of each model and determine the gradient tensor corresponding to the parallel process of the model based on the model state variables.

[0188] The parameter update module is used to update the model parameters of the network model to be updated based on the gradient tensor, so as to obtain the trained network model.

[0189] In one embodiment, the structure partitioning module is further configured to: perform vertical segmentation based on the first linear layer to obtain multiple first-type model components corresponding to the first linear layer, and perform horizontal segmentation based on the second linear layer to obtain multiple second-type model components corresponding to the second linear layer;

[0190] The gradient tensor determination module is further configured to: execute parallel processes for each model corresponding to each first type of model component; perform forward and backward computations for each first type of model component based on the first activation function corresponding to the first linear layer and according to the model state variables to obtain the first output data of each first type of model component; execute parallel processes for each model corresponding to each second type of model component; perform forward and backward computations for each second type of model component based on the second activation function corresponding to the second linear layer and according to the first output data of each first type of model component to obtain the second output data of each second type of model component; and determine each second output data as a gradient tensor corresponding one-to-one with each parallel process of the model.

[0191] In one embodiment, a model parameter updating apparatus is provided, further comprising:

[0192] The training node partitioning module is used to divide the training nodes, which correspond one-to-one with each data parallel process, into multiple pipeline groups; each pipeline group includes at least two training nodes, and the number of training nodes in each pipeline group is the same.

[0193] The average model gradient determination module is used to obtain the order of each training node in the pipeline group, so that each training node performs forward and backward calculations in sequence according to the order to obtain the average model gradient.

[0194] The parameter update module is used to update the model parameters of the network model to be updated based on the average model gradient, so as to obtain the trained network model.

[0195] In one embodiment, the average model gradient determination module is further configured to: divide the full training set corresponding to the network model to be updated into multiple batches of micro-training sets; obtain the order of each training node in the pipeline group; the training nodes include a first training node and a second training node, with the first training node preceding the second training node; the first training node is configured to sequentially obtain each batch of micro-training sets, perform forward computation to obtain the loss value, and send the loss value to the second training node; the second training node is configured to perform backward computation based on the loss value to obtain gradient data; wherein, while the second training node performs backward computation based on the loss value, the first training node obtains the next batch of micro-training sets and performs forward computation in parallel.

[0196] In one embodiment, a model parameter update system is provided, which specifically includes the following components:

[0197] P1 data parallel communication processing component:

[0198] First, the training nodes on the training machine (i.e., the GPUs deployed on the training machine) acquire the model states of the network model to be updated, as well as the tensors corresponding to each model state. They then count the total number of tensors corresponding to each model state and further divide them equally among the data parallel processes based on this total number. Specifically, they optimize the memory usage of ModelStates by distributing the model states across multiple data parallel processes, changing the previous approach of "each data parallel process storing a complete set of model states" to "each data parallel process maintaining only a portion of the model states," thus saving memory.

[0199] Specifically, in the data parallel communication process, the model state variables of the network model are divided among all data parallel processes. Usually, an equal division method is adopted, that is, by obtaining the tensor corresponding to each model state variable, and counting the total number of tensors corresponding to each model state variable, the tensor corresponding to each model state variable is divided equally according to the total number of tensors obtained, and the tensor corresponding to each model state variable is assigned to each data parallel process.

[0200] Similarly, for each data parallel process, a corresponding sub-training dataset is run. By executing each data parallel process, the network model on that data parallel process is updated in parallel based on the sub-training dataset matched to that data parallel process, and gradient reduction processing is performed. Specifically, ReduceScatter processing (i.e., scattering reduction processing) is used for each data parallel process to obtain the reduced gradient corresponding to each data parallel process, that is, to obtain the partial gradient reduction data corresponding to each data parallel process.

[0201] Furthermore, for each data parallel process, its corresponding partial gradient reduction data is processed, which requires allgather processing (i.e., full aggregation processing) to ensure that each data parallel process can obtain the full weight information, thereby achieving parameter updates of the network model based on the full weight information and obtaining a trained network model.

[0202] In one embodiment, the weight of each data-parallel process is determined in the following manner:

[0203] Based on the gradient reduction data corresponding to each data parallel process, the first-order momentum data and second-order momentum data corresponding to each data parallel process are determined respectively; based on the first-order momentum data and the first-order momentum decay coefficient, the first bias correction data of the first-order momentum data is determined, and based on the second-order momentum data and the second-order momentum decay coefficient, the second bias correction data of the second-order momentum data is determined; based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, the weights corresponding to each data parallel process are determined respectively.

[0204] P2 model parallel communication processing component:

[0205] First, the model structure of the network model to be updated is obtained, which may include a first linear layer and a second linear layer. Then, the first linear layer is vertically segmented to obtain multiple first-type model components corresponding to the first linear layer, and the second linear layer is horizontally segmented to obtain multiple second-type model components corresponding to the second linear layer.

[0206] Secondly, by executing the parallel processes of each model corresponding to each first type of model component, and based on the first activation function corresponding to the first linear layer, the forward and backward computations of each first type of model component are performed according to the model state variables to obtain the first output data of each first type of model component.

[0207] Similarly, by executing the parallel processes of each model corresponding to each second type of model component, based on the second activation function corresponding to the second linear layer, and according to the forward and backward computation of each first output data process of the second type of model component, the second output data of each second type of model component is obtained, and then each second output data is determined as a gradient tensor corresponding to each parallel process of the model.

[0208] Furthermore, the gradient tensors corresponding to each parallel process of the model are fully aggregated, meaning that each parallel process of the model can obtain the gradient tensors of other parallel processes of the model. The determined gradient tensors can then be understood as the full gradient tensors. Based on the full gradient tensors, the model structure and model parameters of the model to be updated can be updated to obtain the trained network model.

[0209] P3 pipelined parallel communication processing component:

[0210] First, the training nodes corresponding to each data parallel process are obtained, and the training nodes are divided into multiple pipeline groups. For example, two pipeline groups are set up, and two training nodes are set up in each pipeline group.

[0211] Secondly, the full training set corresponding to the network model to be updated is divided into multiple batches of micro-training sets, and the order of each training node in the pipeline group is obtained. The training nodes include a first training node and a second training node, with the first training node preceding the second training node. The first training node is used to sequentially acquire each batch of micro-training sets, perform forward computation to obtain the loss value, and send the loss value to the second training node. The second training node is used to perform backward computation based on the loss value to obtain gradient data.

[0212] During the reverse calculation based on the loss value at the second training node, the first training node obtains the next batch of micro-training sets and performs forward calculations in parallel.

[0213] Furthermore, for the gradient data obtained from different pipeline groupings, it is necessary to further calculate the leather and model gradients, so as to use the average model gradient as the basis for updating the model parameters of the network model to be updated, thereby realizing the parameter update of the network model and obtaining the trained network model.

[0214] In one embodiment, a specific platform environment for the model parameter update system is provided. The CPU is a 96-core Intel(R) Xeon(R) Platinum 8255C CPU at 2.50GHz, specifically including two physical CPUs, each with 24 cores, for a total of 48 cores and 96 logical CPUs. The GPU is a V100 graphics card with 32GB of RAM and 100Gb bandwidth, accessed via RDMA (Remote Direct Memory Access). In this embodiment, eight training machines can be used, i.e., 64 V100 cards, to train a model with hundreds of billions of parameters, effectively improving the training efficiency of large-scale models.

[0215] The aforementioned model parameter update system includes data parallel communication processing components, model parallel communication processing components, and pipeline parallel communication processing components. This enables multi-level parallel processing of the network model, effectively balancing GPU transmission efficiency and saving GPU memory usage during training. Furthermore, by encapsulating the communication interfaces of the data parallel process group, model parallel process group, and pipeline parallel process group, the interfaces can be directly called, thereby further improving efficiency and model training efficiency.

[0216] Each module in the aforementioned model parameter update device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0217] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as the model state variables of the network model to be updated, data parallelization processes, sub-training datasets, gradients of the network model to be updated on different data parallelization processes, and gradient reduction data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a model parameter update method.

[0218] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0219] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0220] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0221] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0222] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0223] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0224] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0225] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for updating model parameters, characterized in that, The method includes: The model state variables of the network model to be updated are obtained through training nodes on the training machine, and each model state variable is divided into a data parallel process. The training node refers to the GPU card deployed on the training machine. Each GPU card runs one data parallel process. Each data parallel process is set with a portion of the model state variables that need to be saved and updated. Each GPU card is used to update the tensor corresponding to the portion of the model state variables and save the corresponding training results. Based on the model state variables that match each of the data parallel processes, a sub-training dataset corresponding to each of the data parallel processes is determined. Each of the data parallel processes is executed, and the gradient of the network model to be updated on different data parallel processes is determined based on the sub-training datasets corresponding to each of the data parallel processes. In this process, multiple data parallel processes are run simultaneously using multiple GPU cards, and the training results of each sub-training dataset used by each data parallel process are exchanged to obtain the complete training results for the network model to be updated. Gradient reduction processing is performed on each of the gradients to obtain gradient reduction data that matches different parallel data processes; Based on the gradient reduction data, the model parameters of the network model to be updated are updated to obtain the trained network model. The method further includes: performing structural segmentation based on the network model to be updated to obtain multiple model components; the network model to be updated includes a first linear layer and a second linear layer, and the model components include multiple first-type model components corresponding to the first linear layer obtained by vertical segmentation based on the first linear layer, and multiple second-type model components corresponding to the second linear layer obtained by horizontal segmentation based on the second linear layer; dividing each model component into a model parallel process; the model components and the model parallel processes are in one-to-one correspondence; by executing each model parallel process corresponding to each first-type model component, based on the first activation function corresponding to the first linear layer, and according to the model state variables, performing forward computation by applying the identity mapping function, and distributing the same input data to different... Using the same GPU, a backpropagation calculation is performed by applying a full aggregation function. The gradients of each GPU with respect to the input data are summed to obtain a new gradient with respect to the input data, which serves as the first output data for each of the first type of model components. By executing the parallel processes of each model corresponding to each of the second type of model components, and based on the second activation function corresponding to the second linear layer, the forward and backward calculations of the second type of model components are performed according to the first output data and by applying the full aggregation function to obtain the second output data for each of the second type of model components. Each second output data is determined as the gradient tensor corresponding to each of the parallel processes of the models. The model parameters of the network model to be updated are updated according to the gradient tensors to obtain the trained network model.

2. The method according to claim 1, characterized in that, The step of updating the model parameters of the network model to be updated based on the gradient reduction data to obtain the trained network model includes: Based on the gradient reduction data, determine the weights corresponding to the parallel processes of each data; Based on the weights corresponding one-to-one with each of the data parallel processes, full aggregation processing is performed to obtain full weight information. Based on the full weight information, the parameters of the network model to be updated are updated to obtain a trained network model.

3. The method according to claim 1, characterized in that, The execution of each of the data parallel processes, based on a sub-training dataset corresponding to each of the data parallel processes, determines the gradient of the network model to be updated on different data parallel processes, including: Each of the data parallel processes is executed, and the forward computation of the network model to be updated is performed based on the sub-training dataset that corresponds one-to-one with each of the data parallel processes to obtain the training loss value. Based on the training loss value, the reverse calculation of the network model to be updated is performed to obtain the gradient of the network model to be updated in different data parallel processes.

4. The method according to claim 2, characterized in that, The step of determining the weights corresponding to each parallel process based on the gradient reduction data includes: Based on the gradient reduction data corresponding to each of the data parallel processes, the first-order momentum data and the second-order momentum data corresponding to each of the data parallel processes are determined respectively. Based on the first-order momentum data and the first-order momentum decay coefficient, a first bias correction data for the first-order momentum data is determined, and based on the second-order momentum data and the second-order momentum decay coefficient, a second bias correction data for the second-order momentum data is determined. Based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, weights corresponding to the parallel processes of each data are determined.

5. The method according to claim 1, characterized in that, The method further includes: The training nodes, each corresponding to one of the data parallel processes, are divided into multiple pipeline groups; each pipeline group includes at least two training nodes, and the number of training nodes in each pipeline group is the same. Obtain the sorting of each training node in the pipeline group, and perform forward and backward computations sequentially according to the sorting to obtain the average model gradient; Based on the average model gradient, the model parameters of the network model to be updated are updated to obtain the trained network model.

6. A model parameter update device, characterized in that, The device includes: The model state acquisition module is used to acquire the model state of the network model to be updated through the training nodes on the training machine, and to divide each model state into a data parallel process; wherein, the training node refers to the GPU card deployed on the training machine, each GPU card runs one data parallel process, each data parallel process is set with a portion of model state that needs to be saved and updated, and each GPU card is used to update the tensor corresponding to the portion of model state and save the corresponding training results; The sub-training dataset determination module is used to determine the sub-training dataset that corresponds one-to-one with each of the data parallel processes based on the model state variables that match each of the data parallel processes. The gradient determination module is used to execute each of the data parallel processes and determine the gradient of the network model to be updated on different data parallel processes based on the sub-training datasets corresponding to each of the data parallel processes. In this way, multiple data parallel processes are run simultaneously using multiple GPU cards and the GPU cards interact with each other to exchange the training results of each sub-training dataset used by each data parallel process, so as to obtain the complete training result for the network model to be updated. The gradient reduction processing module is used to perform gradient reduction processing on each of the gradients to obtain gradient reduction data that matches different parallel data processes. The model parameter update module is used to update the model parameters of the network model to be updated according to the gradient reduction data, so as to obtain the trained network model. The device further includes: The structure partitioning module is used to perform structure partitioning based on the network model to be updated to obtain multiple model components; the network model to be updated includes a first linear layer and a second linear layer, and the model components include multiple first-type model components corresponding to the first linear layer obtained by vertical partitioning based on the first linear layer, and multiple second-type model components corresponding to the second linear layer obtained by horizontal partitioning based on the second linear layer. A component partitioning module is used to partition each of the model components into a model parallel process; the model components and the model parallel processes correspond one-to-one. The gradient tensor determination module is used to: execute parallel processes for each model corresponding to each of the first type of model components; based on the first activation function corresponding to the first linear layer, perform forward computation by applying an identity mapping function according to the model state variables; distribute the same input data to different GPU cards; perform backward computation by applying a full aggregation function; sum the gradients on each GPU card with respect to the input data to obtain a new gradient with respect to the input data, which serves as the first output data for each of the first type of model components; and execute parallel processes for each model corresponding to each of the second type of model components; based on the second activation function corresponding to the second linear layer, perform forward and backward computations for each of the second type of model components according to the first output data and by applying the full aggregation function to obtain the second output data for each of the second type of model components, and determine each of the second output data as the gradient tensor corresponding to each of the parallel processes. The parameter update module is used to update the model parameters of the network model to be updated based on the gradient tensor, so as to obtain the trained network model.

7. The apparatus according to claim 6, characterized in that, The model parameter update module is also used for: Based on the gradient reduction data, weights corresponding to each parallel process are determined; based on the weights corresponding to each parallel process, full aggregation is performed to obtain full weight information; based on the full weight information, the parameters of the network model to be updated are updated to obtain the trained network model.

8. The apparatus according to claim 6, characterized in that, The gradient determination module is further configured to: Each of the data parallel processes is executed, and the forward computation of the network model to be updated is performed based on the sub-training dataset that corresponds one-to-one with each of the data parallel processes to obtain the training loss value. Based on the training loss value, the reverse calculation of the network model to be updated is performed to obtain the gradient of the network model to be updated in different data parallel processes.

9. The apparatus according to claim 7, characterized in that, The model parameter update module is also used for: Based on the gradient reduction data corresponding to each of the data parallel processes, first-order momentum data and second-order momentum data corresponding to each of the data parallel processes are determined respectively; based on the first-order momentum data and the first-order momentum decay coefficient, first bias correction data of the first-order momentum data is determined, and based on the second-order momentum data and the second-order momentum decay coefficient, second bias correction data of the second-order momentum data is determined; based on the first bias correction data, the second bias correction data, and the learning rate of the network model to be updated, weights corresponding one-to-one with each data parallel process are determined respectively.

10. The apparatus according to claim 6, characterized in that, The model parameter update device further includes: The training node partitioning module is used to partition the training nodes corresponding one-to-one with each of the data parallel processes into multiple pipeline groups; each pipeline group includes at least two training nodes, and the number of training nodes in each pipeline group is the same. The average model gradient determination module is used to obtain the order of each training node in the pipeline group, so that each training node performs forward calculation and backward calculation in sequence according to the order to obtain the average model gradient. The parameter update module is used to update the model parameters of the network model to be updated based on the average model gradient, so as to obtain the trained network model.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images