Gradient communication method, system and device for sparse model parallel training and medium

Abstract

According to the sparse model parallel training-oriented gradient communication method, system and device and the medium provided by the invention, a data parallel mode is used for dense parameters and training data of a language model to be trained on each computing node; a model parallel mode is used for word embedding table parameters of the to-be-trained language model on each computational node, after configuration is completed, the multiple computational nodes conduct iterative training of the to-be-trained language model at the same time according to set training parameters, and dense communication and sparse communication are conducted in each round of iterative training process. Based on the condition that the number of columns of a word embedding surface layer of a natural language processing model is large, in the word embedding surface layer, a data parallel and dense communication mode in the prior art is not adopted any more, a model parallel and sparse communication mode is adopted, and high expansibility is obtained while high communication efficiency is obtained.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a gradient communication method, system, device and medium for parallel training of sparse models. Background Art

[0002] Training deep neural network models (hereinafter referred to as "models") in the field of natural language processing applications is a very time-consuming task. In recent years, with the rapid development of deep learning and artificial intelligence, language models have been increasingly widely used, and the number of parameters and data has also increased. The computing power of stand-alone hardware devices can no longer meet the computing requirements of the model, and it is necessary to rely on distributed parallel means for training.

[0003] Parallel training refers to the method of using multiple computing devices, such as graphics cards, to jointly train a model. Multiple devices can enable the training process to be carried out in parallel, shortening the training cycle. At present, some relatively mature distributed parallel training technologies have emerged. In the current parallel training system, data parallelism is the most commonly used distributed parallel training method. However, in data parallel training, due to the sparsity of the gradient of the word embedding table of the natural language processing model, sparse gradient communication becomes a performance bottleneck, and the throughput of parallel training is severely limited.

[0004] At present, with the continuous expansion of the scale of natural language processing models and the increase in data volume, traditional single-machine training can no longer meet the needs of large-scale models. In order to speed up the training process and make full use of computing resources, data parallel training technology has emerged. However, the communication of sparse gradients in the model in data parallelism still has the problem of not being able to balance efficiency and scalability, which affects the training efficiency of natural language processing models.

[0005] Therefore, it is an urgent problem to be solved by those skilled in the art to provide a gradient communication method, system, device and medium for parallel training of sparse models to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a gradient communication method for parallel training of sparse models. The method has clear logic, is safe, effective, reliable and easy to operate, can effectively improve the communication efficiency of sparse gradients in the model and maintain high scalability.

[0007] Based on the above objectives, the technical solution provided by the present invention is as follows: A gradient communication method for parallel training of sparse models includes the following steps: Provide the language model to be trained, training data, multiple computing nodes, and training parameters; Using data parallelism on dense parameters of the language model to be trained and the training data on each of the computing nodes; Using model parallelism for word embedding table parameters of the language model to be trained on each of the computing nodes; The plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform dense communication and sparse communication during each round of iterative training.

[0008] Preferably, it also includes: allocating a number of computing devices to each of the computing nodes; The training parameters include: the total number N of all computing devices and the batch size of training data for each round of iteration.

[0009] Preferably, the dense parameters of the language model to be trained on each of the computing nodes and the training data are processed in data parallelism, comprising the following steps: The language model to be trained is replicated in a corresponding number according to the total number N of all computing devices, so that each computing device on each computing node corresponds to one language model to be trained, and the same dense parameters are configured for each computing device; The training data is divided into multiple batches according to the batch size of the training data in each round of iteration, and the training data of the current batch is input into each of the computing devices, and each of the computing devices runs in parallel.

[0010] Preferably, the plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform dense communication during each round of iterative training, including the following steps: The plurality of computing nodes simultaneously iteratively train the language model to be trained according to the dense parameters; During each round of iterative training, each computing device performs forward calculation and backward calculation of the language model to be trained according to the current batch of training data, and performs gradient summary update and dense communication synchronization between multiple computing devices through the first communication method.

[0011] Preferably, the dense communication process also includes communication scheduling, specifically: According to the dependency relationship of the language model to be trained and in the order in which the dense layers of the language model to be trained perform forward calculations, each dense layer of the language model to be trained is assigned a priority from high to low, and the communication scheduling is achieved through the priority.

[0012] Preferably, the word embedding table parameters of the language model to be trained on each computing node are processed using model parallelism, comprising the following steps: According to the word embedding table parameters, the word embedding table is divided into N parts along the matrix columns according to the total number of all computing devices, and each computing device is sequentially configured with each divided word embedding table having the same matrix size; The training data is input into each of the computing devices, and each of the computing devices runs in parallel.

[0013] Preferably, the plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform sparse communication during each round of iterative training, including the following steps: The plurality of computing nodes simultaneously iteratively train the language model to be trained according to each of the segmented word embedding tables having the same matrix size; During each round of iterative training, each computing device performs forward calculation and backward calculation of the language model to be trained according to the training data, and merges the calculation results and synchronizes sparse communication between multiple computing devices through a second communication method.

[0014] Preferably, the second communication method is specifically: all-to-all aggregate communication; The all-to-all aggregate communication comprises the following steps: After each of the computing devices performs forward calculation of the language model to be trained according to the training data, performing a first full-to-full aggregate communication on the result of the forward calculation; After each of the computing devices performs reverse calculation of the language model to be trained according to the training data, a second full-to-full aggregation communication is performed on the sparse gradient results of the forward calculation.

[0015] Preferably, the sparse communication process also includes communication scheduling, specifically: According to the dependency relationship of the language model to be trained, and in accordance with the order in which the word embedding surfaces of the language model to be trained are forward calculated, priority is assigned to the word embedding surfaces of the language model to be trained from high to low; The sparse gradient results of the forward calculation are divided into the sparse gradient results used in the next round of iteration and the sparse gradient results not used in the next round of iteration, and the communication is assigned the highest priority and the lowest priority respectively; The result of the forward calculation is assigned the second highest priority to the communication to implement the communication scheduling.

[0016] A gradient communication system for parallel training of sparse models, comprising: A parameter module is used to provide a language model to be trained, training data, multiple computing nodes, and training parameters; A data parallel module, used for using data parallelism for dense parameters of the language model to be trained and the training data on each of the computing nodes; A model parallel module, used for using model parallelism for the word embedding table parameters of the language model to be trained on each of the computing nodes; The iterative training module is used for the plurality of computing nodes to simultaneously perform iterative training of the language model to be trained according to the training parameters, and to perform dense communication and sparse communication during each round of iterative training.

[0017] An electronic device, comprising: Memory, storing programs or instructions; A processor, executing the program or instructions; Wherein, when executing the program or instruction, the processor implements the steps of the gradient communication method for parallel training of sparse models as described in any of the above items.

[0018] A readable storage medium having a program or instruction stored thereon, wherein when the program or instruction is executed by a processor, the steps of the gradient communication method for parallel training of sparse models as described in any one of the above items are implemented.

[0019] The gradient communication method for parallel training of sparse models provided by the present invention uses data parallelism for dense parameters and training data of the language model to be trained on each computing node, and uses model parallelism for word embedding table parameters of the language model to be trained on each computing node. After configuration is completed, multiple computing nodes simultaneously perform iterative training of the language model to be trained according to the set training parameters, and perform dense communication and sparse communication in each round of iterative training.

[0020] Based on the fact that the word embedding table of the natural language processing model has a large number of columns, the data parallelism plus dense communication method in the existing technology is no longer used in the word embedding table. Instead, the model parallelism plus sparse communication method is adopted to achieve high communication efficiency while obtaining higher scalability.

[0021] The present invention also provides a gradient communication system, electronic device and storage medium for parallel training of sparse models. Since they belong to the same technical concept as this method, solve the same technical problem, and should have the same beneficial effects, they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flowchart of a gradient communication method for parallel training of sparse models provided by an embodiment of the present invention; Figure 2 A flowchart of step S2 provided in an embodiment of the present invention; Figure 3 A flowchart of step S3 provided in an embodiment of the present invention; Figure 4 A flow chart of all-to-all aggregate communication provided by an embodiment of the present invention; Figure 5 A communication scheduling flow chart for a sparse communication process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The embodiments of the present invention are written in a progressive manner.

[0026] The embodiments of the present invention provide a gradient communication method, system, device and medium for parallel training of sparse models, which mainly solves the technical problem in the prior art that the communication of sparse gradients in the model still cannot take into account both efficiency and scalability.

[0027] like Figure 1 As shown, a gradient communication method for parallel training of sparse models includes the following steps: S1. Provide the language model to be trained, training data, multiple computing nodes and training parameters; S2. Using data parallelism for the dense parameters and training data of the language model to be trained on each computing node; S3. Use model parallelism for the word embedding table parameters of the language model to be trained on each computing node; S4. Multiple computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform dense communication and sparse communication during each round of iterative training.

[0028] In step S1, a deep neural network model of a natural language processing application field to be trained, multiple training samples, the number of computing nodes and training parameters are provided; In step S2 and step S3, the dense parameters of the model to be trained on each computing node and the training data are aligned in a data-parallel training mode; the sparse parameters (i.e., word embedding table parameters) of the model to be trained on each computing node are configured in a model-parallel training mode; In step S4, after the configuration is completed, multiple computing nodes simultaneously perform iterative training of the model to be trained according to the training parameters, and perform dense communication and sparse communication during each round of iterative training.

[0029] Preferably, it also includes: allocating a number of computing devices to each computing node respectively; The training parameters include: the total number of computing devices N and the batch size of training data for each iteration.

[0030] In the actual application process, after step S1, it also includes allocating a number of computing devices to each computing node; the sum of all computing devices is N, and the batch size of each round of iterative training data is set to B.

[0031] like Figure 2 As shown, preferably, step S2 includes the following steps: A1. Copy the language model to be trained in the number corresponding to the total number N of all computing devices, so that each computing device on each computing node corresponds to a language model to be trained, and configure the same dense parameters for each computing device; A2. Divide the training data into multiple batches according to the batch size of each round of iteration training data, and input the training data of the current batch into each computing device, and each computing device runs in parallel.

[0032] In step A1 to step A2, the model to be trained is copied N times according to the total number of all computing devices, and each computing device on each computing node corresponds to a model to be trained. At the same time, each computing device is configured with the same dense parameters, and the training data is divided into multiple batches according to the batch size B of the training data in each round of iteration, and the training data of the current batch is input into each computing device, and each computing device runs in parallel; It should be noted that in data parallelism, multiple devices process different data batches at the same time, and the model parameters on each device remain the same. Data parallelism is suitable for processing large data sets. Each device performs calculations on different data batches and updates the shared model parameters by averaging gradients. This helps speed up training, especially on large-scale data sets. The challenge of data parallelism lies in how to efficiently transmit model parameters and gradients between devices, and ensure synchronous update of parameters. Some optimization techniques, such as gradient accumulation and asynchronous updates, can help in data parallelism.

[0033] Preferably, step S4, performing dense communication in each round of iterative training, includes the following steps: Multiple computing nodes simultaneously iteratively train the language model to be trained based on dense parameters; During each round of iterative training, each computing device performs forward calculation and backward calculation of the language model to be trained according to the current batch of training data, and performs gradient summary update and dense communication synchronization between multiple computing devices through the first communication method.

[0034] In the actual application process, each computing node performs iterative training on the model to be trained according to the dense parameters and then summarizes them; in each round of iterative training, the forward and backward calculations of the model to be trained are performed according to the input current batch of training data, and the gradient summary and update as well as the dense communication synchronization between multiple computing devices are performed through the AllReduce communication method; It should be noted that the Allreduce operation is the most important operation in aggregate communication. It collects vectors from all hosts, "aggregates" the vectors in an element-wise manner, and broadcasts the aggregated results to them ("aggregation": summing, or finding the maximum / minimum value, etc.); a large number of Allreduce algorithms have emerged in the past few decades, among which the classic algorithms include the recursive doubling algorithm, the ring algorithm, the Rabenseifner algorithm, etc.

[0035] In this embodiment, a recursive doubling algorithm is selected for gradient summary update and dense communication / state synchronization between multiple computing devices.

[0036] Preferably, the dense communication process also includes communication scheduling, specifically: According to the dependency relationship of the language model to be trained and the order in which the dense layers of the language model to be trained are forward calculated, each dense layer of the language model to be trained is assigned a priority from high to low, and communication scheduling is achieved through the priority.

[0037] In actual application, communication scheduling is also required during dense communication. Specifically, the deep neural network model is represented in the form of a directed acyclic graph. According to the order of forward calculation of each dense layer of the model, a priority is given to the dense gradient communication operation of each dense layer of the model. The earlier the forward calculation of the corresponding model is performed (i.e., the shallower the model layer), the higher the priority. When there are multiple data that need to be communicated, the communication scheduling method of the first-in-first-out queue is no longer used. The communication scheduling method of the priority queue is used, that is, the communication with a higher priority is given priority. Then, the shallow model can complete the gradient communication earlier. After the first layer of the model (with the highest communication priority) is completed, the next round of iteration can be carried out. At this time, some gradient communication operations with lower priorities are still in progress, so some gradient communications can overlap with the forward calculation of the next round of model training iteration, and the purpose of overlapping communication with more computing operations is initially achieved.

[0038] like Figure 3 As shown, preferably, step S3 includes the following steps: B1. Divide the word embedding table into N parts along the matrix columns according to the total number of all computing devices according to the word embedding table parameters, and sequentially configure each computing device with a word embedding table of the same matrix size; B2. Input the training data into each computing device, and each computing device runs in parallel.

[0039] In step B1 to step B2, the word embedding table is cut into N parts along the matrix column according to the secondary embedding table parameters, each computing device is sequentially configured with each of the cut word embedding tables with the same matrix size but different contents, the training data is input into each computing device, and each computing device runs in parallel; It should be noted that in model parallelism, a large neural network model is divided into multiple parts, each of which runs on different devices (such as different GPUs or servers). Each device is responsible for calculating a part of the model and passing its results to other devices, which are finally merged to obtain the overall model output. Model parallelism is mainly used to deal with situations where the model is too large to be fully loaded into the memory of a single device. The challenge of model parallelism is to effectively divide the model and manage the communication between the parts. This usually requires careful design and organization of the model architecture in order to reasonably distribute the computing load and reduce communication overhead; In this embodiment, the parameters of the word embedding table are a matrix of size VxH, and each row of the matrix corresponds to a word vector. At this time, the model parallel method is used to divide the word embedding table into N parts along the matrix columns, and the number of matrix columns in each part is D=H / N. Each computing device keeps one of the parts, that is, the parameters of the word embedding table of size VxD, with different contents. The parts other than the word embedding table of the model maintain the data parallel method, which constitutes a hybrid parallel training method. Preferably, step S4, performing sparse communication in each round of iterative training, includes the following steps: Multiple computing nodes simultaneously iteratively train the language model to be trained based on each split word embedding table with the same matrix size; During each round of iterative training, each computing device performs forward and backward calculations of the language model to be trained according to the training data, and merges the calculation results and synchronizes sparse communications between multiple computing devices through the second communication method.

[0040] During actual application, each computing node iteratively trains the language model to be trained based on each segmented word embedding table with the same matrix size but different content, and each computing device is responsible for a part of the model. In each round of iterative training, each computing device performs forward and backward calculations on the language model to be trained based on the training data, and merges the calculation results of each computing device and synchronizes sparse communications between multiple computing devices through the AllToAll communication method.

[0041] Preferably, the second communication method is specifically: all-to-all aggregate communication; like Figure 4 As shown, full-to-full aggregation communication includes the following steps: C1. After each computing device performs forward calculation of the language model to be trained according to the training data, the first full-to-full aggregate communication is performed on the results of the forward calculation; C2. After each computing device performs reverse calculation of the language model to be trained based on the training data, a second full-to-full aggregation communication is performed on the sparse gradient results of the forward calculation.

[0042] In step C1 to step C2, the calculation of the word embedding table and the gradient communication are performed in the following manner: after the word embedding table completes the forward calculation, an AllToAll communication is performed on the result of the forward calculation; after the word embedding table completes the reverse calculation, an AllToAll communication is performed on the sparse gradient result of the forward calculation; It should be noted that the AllToAll operation of OpenMPl is a communication operation in parallel computing, which is used to send data from all sending processes to all receiving processes. In the AllToAll operation, each sending process sends the data in its sending buffer to all other receiving processes, and each receiving process receives data from all sending processes. This operation can be used to implement some parallel computing algorithms, such as matrix multiplication, set operations, etc. The AllToAll operation of OpenMPl uses a communication protocol based on message passing, and a message is passed between each sending process and receiving process. In order to achieve efficient AllToAll operation, OpenMPI uses some optimization techniques, such as binary data transmission, multi-threaded processing, etc. In specific implementation, the AllToAll operation of OpenMPl can be divided into the following steps: 1. The sending process sends data to all receiving processes, and the receiving process waits to receive data. 2. Messages are passed between the sending process and the receiving process, and the receiving process receives data from the sending process. 3. After all sending processes and receiving processes have completed message passing, the AllToAll operation ends.

[0043] In this embodiment, when the above steps are used for communication, the communication volume of each word embedding table AllToAll is (N-1)BH / N, so the total communication volume for sparse communication of the word embedding table in each iteration is 2(N-1)BH / N, while the communication volume of the AllReduce communication method is 2(N-1)VH / N, and the communication volume of the AllGather communication method is (N-1)BH. Since B is much smaller than V, and N is generally a large value, the above communication method has a smaller communication volume, which can improve the efficiency of communication in data parallelism. At the same time, the total amount of communication will not increase significantly with the increase of N, and it has good scalability. The hybrid parallel method in the embodiment can effectively combine the communication advantages of AllToAll and AllReduce, and can effectively improve the communication efficiency compared to the model parallel method that uses AllToAll communication and the data parallel method that uses AllReduce communication.

[0044] like Figure 5 As shown, preferably, in the process of sparse communication, communication scheduling is also included, specifically: D1. According to the dependency relationship of the language model to be trained, and in the order of forward calculation of each word embedding surface of the language model to be trained, the word embedding surface of the language model to be trained is given priority from high to low; D2. Divide the sparse gradient results of the forward calculation into the sparse gradient results that have been used in the next round of iteration and the sparse gradient results that have not been used in the next round of iteration, and assign the highest communication priority and the lowest communication priority respectively; D3. Assign the result of the forward calculation to the second highest priority of communication to achieve communication scheduling.

[0045] In step D1 to step D3, the word embedding table is often located in the first layer of the model and has the highest priority. The next round of iteration can only be started when the sparse communication of the word embedding table is completed, so as to achieve the purpose of overlapping the communication and forward calculation through the priority scheduling queue. In order to reduce the sparse communication overhead of the word embedding table, the priority of the word embedding table can use a finer granularity, because the communication of the word embedding table is sparse, that is, the corresponding vector of the data used in the current iteration, then what is really needed in the next round of iteration is not the complete sparse gradient communication, but the corresponding vector gradient of the data used in the next round of iteration. Based on this fact, the sparse gradients of the word embedding table generated in the current iteration are divided into two categories: the corresponding data used in the next round of iteration and the data not used in the next round of iteration. And the two types of sparse gradients are separated, and the corresponding sparse gradients used in the next round of iteration are given the highest communication priority, and the corresponding sparse gradients not used in the next round of iteration are given the lowest communication priority. In this case, the necessary communication volume required before the start of the next round of iteration can be further reduced, and more communication operations can be overlapped with the forward calculation of the next round of model training iteration; In this embodiment, according to the dense communication scheduling method, the word embedding surface layer is also given priority. On the basis of using the priority communication scheduling queue, each sparse communication in the model is further analyzed, and the second AllToAll sparse gradient communication in step C2 is split into two communications, and the highest priority and the lowest priority are respectively given, and the first sparse output communication AllToAll in step C1 is given the second highest priority. Communication calculation overlap is performed as much as possible to further improve the efficiency of parallel training. For example, for a gradient of size G and sparsity a, where a is greater than 0 and less than 1, the use of AllToAll sparse gradients can reduce the communication overhead to aG. In the sparse gradients, the parameters corresponding to the gradients with a ratio of b (b is greater than 0 and less than 1) are used in the next round of iterations, that is, the parameters corresponding to the gradients with a ratio of 1-b are not used in the next round of iterations. Priority communication is given to the communication required in the next round of iterations, and the communication overhead can be reduced to abG.

[0046] A gradient communication system for parallel training of sparse models, comprising: A parameter module is used to provide a language model to be trained, training data, multiple computing nodes, and training parameters; A data parallel module, used for using data parallelism for dense parameters of the language model to be trained and the training data on each of the computing nodes; A model parallel module, used for using model parallelism for the word embedding table parameters of the language model to be trained on each of the computing nodes; The iterative training module is used for the plurality of computing nodes to simultaneously perform iterative training of the language model to be trained according to the training parameters, and to perform dense communication and sparse communication during each round of iterative training.

[0047] An electronic device, comprising: Memory, storing programs or instructions; A processor, executing the program or instructions; Wherein, when executing the program or instruction, the processor implements the steps of the gradient communication method for parallel training of sparse models as described in any of the above items.

[0048] A readable storage medium having a program or instruction stored thereon, wherein when the program or instruction is executed by a processor, the steps of the gradient communication method for parallel training of sparse models as described in any one of the above items are implemented.

[0049] In actual application, a gradient communication system, electronic device and storage medium for parallel training of sparse models are also disclosed. The communication system, the processor in the electronic device and the storage medium are all used to implement the steps of the gradient communication method for parallel training of sparse models, which will not be repeated here.

[0050] In the embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0051] In addition, all functional modules in the embodiments of the present invention may be integrated into one processor, or each module may be a separate device, or two or more modules may be integrated into one device; each functional module in the embodiments of the present invention may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0052] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, the steps of the above method embodiment are executed; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.

[0053] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0054] In addition, in the description of the embodiments of the present application, “plurality” refers to two or more than two.

[0055] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0056] If a flow chart is used in the present application, the flow chart is used to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or a certain step or several steps of operations can be removed from these processes.

[0057] The above is a detailed introduction to a gradient communication method, system, device and medium for parallel training of sparse models provided by the present invention. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined in this article can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this article, but will conform to the widest scope consistent with the principles and novel features disclosed in this article.

Claims

1. A gradient communication method for parallel training of sparse models, characterized in that: The steps include: Provide the language model to be trained, training data, multiple computing nodes, and training parameters; Using data parallelism on dense parameters of the language model to be trained and the training data on each of the computing nodes; Using model parallelism for word embedding table parameters of the language model to be trained on each of the computing nodes; The plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform dense communication and sparse communication during each round of iterative training.

2. The gradient communication method for parallel training of sparse models according to claim 1, characterized in that: Also includes: Allocating a number of computing devices to each of the computing nodes respectively; The training parameters include: the total number N of all computing devices and the batch size of training data for each round of iteration.

3. The gradient communication method for parallel training of sparse models according to claim 2, characterized in that: The method of using data parallelism for the dense parameters of the language model to be trained and the training data on each computing node comprises the following steps: The language model to be trained is replicated in a corresponding number according to the total number N of all computing devices, so that each computing device on each computing node corresponds to one language model to be trained, and the same dense parameters are configured for each computing device; The training data is divided into multiple batches according to the batch size of the training data in each round of iteration, and the training data of the current batch is input into each of the computing devices, and each of the computing devices runs in parallel.

4. The gradient communication method for parallel training of sparse models according to claim 3, characterized in that: The plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform dense communication during each round of iterative training, including the following steps: The plurality of computing nodes simultaneously iteratively train the language model to be trained according to the dense parameters; During each round of iterative training, each computing device performs forward calculation and backward calculation of the language model to be trained according to the current batch of training data, and performs gradient summary update and dense communication synchronization between multiple computing devices through the first communication method.

5. The gradient communication method for parallel training of sparse models according to claim 4, characterized in that: The dense communication process also includes communication scheduling, specifically: According to the dependency relationship of the language model to be trained and in the order in which the dense layers of the language model to be trained perform forward calculations, each dense layer of the language model to be trained is assigned a priority from high to low, and the communication scheduling is achieved through the priority.

6. The gradient communication method for parallel training of sparse models according to claim 2, characterized in that: The method of using model parallelism for the word embedding table parameters of the language model to be trained on each computing node comprises the following steps: According to the word embedding table parameters, the word embedding table is divided into N parts along the matrix columns according to the total number of all computing devices, and each computing device is sequentially configured with each divided word embedding table having the same matrix size; The training data is input into each of the computing devices, and each of the computing devices runs in parallel.

7. The gradient communication method for parallel training of sparse models according to claim 6, characterized in that: The plurality of computing nodes simultaneously perform iterative training of the language model to be trained according to the training parameters, and perform sparse communication during each round of iterative training, including the following steps: The plurality of computing nodes simultaneously iteratively train the language model to be trained according to each of the segmented word embedding tables having the same matrix size; During each round of iterative training, each computing device performs forward calculation and backward calculation of the language model to be trained according to the training data, and merges the calculation results and synchronizes sparse communication between multiple computing devices through a second communication method.

8. The gradient communication method for parallel training of sparse models according to claim 7, characterized in that: The second communication method is specifically: all-to-all aggregate communication; The all-to-all aggregate communication comprises the following steps: After each of the computing devices performs forward calculation of the language model to be trained according to the training data, performing a first full-to-full aggregate communication on the result of the forward calculation; After each of the computing devices performs reverse calculation of the language model to be trained according to the training data, a second full-to-full aggregation communication is performed on the sparse gradient results of the forward calculation.

9. The gradient communication method for parallel training of sparse models according to claim 8, characterized in that: The sparse communication process also includes communication scheduling, specifically: According to the dependency relationship of the language model to be trained, and in accordance with the order in which the word embedding surfaces of the language model to be trained are forward calculated, priority is assigned to the word embedding surfaces of the language model to be trained from high to low; The sparse gradient results of the forward calculation are divided into the sparse gradient results used in the next round of iteration and the sparse gradient results not used in the next round of iteration, and the communication is assigned the highest priority and the lowest priority respectively; The result of the forward calculation is assigned the second highest priority to the communication to implement the communication scheduling.

10. A gradient communication system for parallel training of sparse models, characterized in that: include: A parameter module is used to provide a language model to be trained, training data, multiple computing nodes, and training parameters; A data parallel module, used for using data parallelism for dense parameters of the language model to be trained and the training data on each of the computing nodes; A model parallel module, used for using model parallelism for the word embedding table parameters of the language model to be trained on each of the computing nodes; The iterative training module is used for the plurality of computing nodes to simultaneously perform iterative training of the language model to be trained according to the training parameters, and to perform dense communication and sparse communication during each round of iterative training.

11. An electronic device, characterized in that: include: Memory, storing programs or instructions; A processor, executing the program or instruction; Wherein, when executing the program or instruction, the processor implements the steps of the gradient communication method for parallel training of sparse models as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the gradient communication method for parallel training of sparse models as described in any one of claims 1 to 9 are implemented.

Images