Large model parallel training communication optimization method and device

Abstract

A large model parallel training communication optimization method is applied to a system composed of a plurality of computing devices, each computing device is provided with a GPU, the GPU comprises at least one process, the process is used for a task partition in large model parallel training, the method is used for optimizing a ZeRO-1 stage of ZeRO data, and the method comprises the steps that forward calculation and reverse calculation of a model are executed, gradient information is determined; updating the momentum in the local optimizer state based on the gradient information; the updated momentum is processed through 1-bit Allreduce communication, and errors generated in the compression process are updated; a model parameter partition stored on each computing device is updated using the aggregated momentum and variance. The method can reduce the communication overhead and improve the parallel training efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of large model training, and in particular to a large model parallel training communication optimization method. Background Art

[0002] Traditional data parallelism statically maintains all model states required for the entire training process in all data parallel processes, resulting in memory redundancy. ZeRO data parallelism eliminates memory redundancy in data parallel processes by partitioning the model state instead of copying it, so that the memory consumption on each computing device is inversely proportional to the data parallel dimension. ZeRO data parallelism has therefore become popular in academia and industry. ZeRO has three main optimization stages, which store the optimizer state, gradients, and model parameters on multiple computing devices, reducing the memory required to store these states on each computing device in traditional data parallelism. However, this brings more communication overhead during training. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product including a computer program for optimizing communication of large model parallel training, which can reduce communication overhead and improve the efficiency of parallel training.

[0004] In the first aspect, an embodiment of the present application provides a large model parallel training communication optimization method, which is applied to a system composed of multiple computing devices, each computing device is configured with a GPU, the GPU includes at least one process, the process is used for a task partition in the large model parallel training, and the method is used to optimize the ZeRO-1 stage of ZeRO data. The method includes: performing forward and reverse calculations of the model to determine gradient information; based on the gradient information, updating the momentum in the local optimizer state; processing the updated momentum through 1-bit Allreduce communication, and updating the error generated during the compression process; using the aggregated momentum and variance to update the model parameter partition stored on each computing device.

[0005] In some possible implementations, 1-bit Allreduce communication includes: using the All-to-All communication primitive to send compressed momentum data blocks to other computing devices and receive compressed momentum data blocks from other computing devices; performing reduction operations on all received momentum data blocks; and using the Allgather communication primitive to collect momentum data blocks from all computing devices to complete global aggregation of momentum.

[0006] In some possible implementations, updating the error generated during the compression process includes: recording the error generated during each compression, and using the error to compensate for the momentum after compression in the next iteration.

[0007] In some possible implementations, the method is applied to the optimization of the ZeRO-2 stage of ZeRO data. After determining the gradient information, the method further includes: using 1-bit Reduce-Scatter for communication to determine the aggregated momentum; and completing the update of the model parameters based on the aggregated momentum and the fixed variance.

[0008] In some possible implementations, the method further includes: communicating the model parameters via Allgather.

[0009] In some possible implementations, the method is applied to the optimization of the ZeRO-3 stage of ZeRO data, and the method includes: each process performs forward calculation of the model and uses the Allgather communication operation to collect complete model information on all computing devices; after completing the forward calculation, each computing device discards the part that does not belong to the current parameter partition; performs reverse calculation of the model, determines gradient information, and uses the Allgather communication operation to collect gradient information on all computing devices; performs 1-bit Reduce-Scatter communication and updates the compression error; the computing device discards the parameter information of non-current partitions again and only retains the necessary data of the current partition; uses the aggregated gradient information and variance to calculate each process to independently update its assigned model parameter partition.

[0010] In the second aspect, an embodiment of the present application provides a large-model parallel training communication optimization device, which is deployed in a system composed of multiple computing devices, each computing device is configured with a GPU, the GPU includes at least one process, the process is used for a task partition in the large-model parallel training, and the device is used to optimize the ZeRO-1 stage of ZeRO data. The device includes: a gradient information acquisition module, which is used to perform forward and reverse calculations of the model and determine gradient information; a momentum update module, which is used to update the momentum in the local optimizer state based on the gradient information; a processing module, which is used to process the updated momentum through 1-bit Allreduce communication and update the error generated during the compression process; the processing module is also used to use the aggregated momentum and variance to update the model parameter partitions stored on each computing device.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in any one of the first aspects.

[0012] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method described in any one of the first aspects is executed.

[0013] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program. When the computer program product runs on a processor, the processor executes the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0015] Figure 1 It is a schematic diagram of a 1-bit Allreduce communication method provided in an embodiment of the present application;

[0016] Figure 2 It is a flowchart of a large model parallel training communication optimization method based on 1-bit compression provided in an embodiment of the present application;

[0017] Figure 3 This is a schematic diagram of an optimized process of ZeRO-1 provided in an embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of an optimized process of ZeRO-2 provided in an embodiment of the present application;

[0019] Figure 5 This is a schematic diagram of an optimized process of ZeRO-3 provided in an embodiment of the present application;

[0020] Figure 6 It is a structural diagram of a large model parallel training communication optimization device based on 1-bit compression provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0023] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0024] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0026] To facilitate the understanding of the embodiments of the present application, the following will further explain and illustrate specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0027] First, the technical terms involved in the embodiments of the present application are introduced:

[0028] 1. Communication primitives are basic operations used to describe communication within a group of nodes in distributed computing. These operations cannot be interrupted during execution to ensure the correctness of the operation and the stability of the system. Communication primitives are usually implemented by program segments consisting of several instructions to implement a specific function, such as broadcasting, collection, and protocol. These primitives are the basis of collective communication, and their combination can realize more complex communication modes.

[0029] 2. All-to-All: This is a many-to-many communication primitive where each node in the cluster sends data to all other nodes.

[0030] 3. All-Gather collects the data on each GPU to all GPUs so that all GPUs have a complete set of data.

[0031] 4.Reduce communication is a communication primitive widely used in distributed computing. It involves reduction operations on multiple data sources, such as summation and maximum value, to reduce the amount of data or extract useful information.

[0032] 5. Scatter communication is to send different data on one GPU to different GPUs.

[0033] Next, the technical solution provided by the embodiments of the present application is introduced.

[0034] ZeRO, the full name of which is Zero Redundancy Optimizer, is an optimization technology for distributed training of large-scale models. This technology mainly targets the problem of video memory usage in distributed training. It reduces redundancy by slicing to achieve the effect of time-for-space, thereby allowing larger models to be trained on limited hardware resources. ZeRO is divided into three levels: ZeRO-1, ZeRO-2, and ZeRO-3. ZeRO-1 splits the optimizer state, reducing video memory consumption by 4 times, but the communication volume is the same as data parallelism. ZeRO-2 is based on ZeRO-1 and further splits the gradient, reducing video memory consumption by 8 times, while the communication volume remains unchanged. ZeRO-3 is based on ZeRO-2 and also splits the model parameters, so that the video memory occupied by the model is linearly inversely proportional to the degree of parallelism of data parallelism, but the communication volume will increase. ZeRO technology can reduce memory usage by reducing the number of complete model states that each GPU needs to store, making it possible to train large models. For example, for a model with 7.5B parameters, using ZeRO-1 can reduce memory usage from 120GB to 31.4GB, while ZeRO-2 can further reduce it to 16.6GB. ZeRO-3 is further optimized to allow larger models to be trained on more GPUs, for example, models with more than 1 trillion parameters can be trained on 1024 GPUs. However, this greatly increases communication overhead.

[0035] In view of this, an embodiment of the present application provides a large model parallel training communication optimization method based on 1-bit compression. The 1-bit compression algorithm is a data compression technology that greatly reduces the required storage space or communication bandwidth by representing the data in 1-bit form. In the context of deep learning model training, especially in distributed training, the 1-bit compression algorithm can be used to optimize communication efficiency. The 1-bit compression algorithm based on error compensation means that the data is first compressed into a 1-bit representation, and then the compression error is calculated and stored, and finally the data is used for calculation in the next iteration. When the error is added together, the calculation is performed. In the Adam optimizer, the communication efficiency is improved by compressing the optimizer state (momentum) to 1 bit. The algorithm has achieved good results in the deep learning model training of the traditional data parallel method of the parameter server architecture. However, in the current large language model parallel training, the parameter server architecture can no longer meet the demand for computing efficiency of the growing scale of model training, and the Allreduce architecture has gradually become the mainstream method in large language model training. The large model parallel training communication optimization method based on 1-bit compression provided in the embodiment of the present application is intended to improve the parallel training efficiency by reducing the communication volume and number of communications in ZeRO data parallel training. Specifically, based on the 1-bit compression algorithm, the present invention develops a 1-bit Allreduce communication method suitable for the Allreduce data parallel architecture, which greatly improves the communication efficiency in large model data parallel training; the 1-bit Allreduce method is used to improve the ZeRO data parallel workflow, reduce the number of communications and the amount of communication in parallel training, greatly improve the computational efficiency and training throughput of ZeRO data parallel training, and at the same time, combined with the characteristics of the Adam optimizer, ensure the convergence and accuracy of model training.

[0036] The 1-bit compression algorithm is a data compression technique that significantly reduces the required storage space or communication bandwidth by representing data in 1-bit form. In the context of deep learning model training, especially in distributed training, the 1-bit compression algorithm can be used to optimize communication efficiency. The implementation of the 1-bit Allreduce communication method is based on NCCL (NVIDIA Collective Communications Library) through the two communication primitives All-to-All and Allgather. NCCL is a communication library for high-performance parallel computing. It provides a variety of low-level communication primitives to support efficient data parallel operations on GPUs. There can be multiple processes on the GPU, and the processes participate in the optimization of the model. The execution steps of the method include the following 3 steps:

[0037] S11: For the momentum to be communicated, the error obtained in the last iteration is used to compensate for the error, and then the compression operator is used for compression. Then, the error of the first communication is updated in the second line. The momentum is divided into N blocks in the third line, and All-to-All is used for communication.

[0038] In this embodiment, the momentum in the optimizer state is compressed to 1-bit representation. Since 1-bit compression will lose the numerical information of the data, it is necessary to calculate the error introduced in the compression process and store it. In subsequent iterations, this error value is used to compensate the compressed data to reduce the impact of compression. In each iteration, the compressed data and the stored error are used for calculation together. Before the communication or update operation, the error is added to the data represented by 1-bit to restore part of the original data information and complete the error compensation. In the current step, the compression error calculated in the last iteration is first used to compensate the current momentum. After the error compensation is completed, the momentum is compressed to 1-bit representation using the compression operator. After completing the 1-bit compression, it is necessary to calculate the new error introduced by this compression and store it for use in the next iteration. The compressed momentum data is divided into N blocks, where N is the number of computing devices participating in the communication. Using the All-to-All communication mode, each computing device sends its own data block to all other devices. In All-to-All communication, each i-th worker process passes blocks other than the i-th block to the corresponding process, and receives blocks passed by processes other than the i-th worker process. Figure 1 FIG. 1 is a schematic diagram showing a 1-bit Allreduce communication method provided in an embodiment of the present application. Figure 1 As shown in the figure, in the All-to-All communication, take 3 processes as an example, each process contains 3 blocks. The first process passes the 2nd block to the second process, and the 3rd block to the third process. At the same time, it receives the 1st block of the second process and the 1st block of the third process. Similarly, the second process passes the 1st block to the first process, passes the 3rd block to the third process, and receives the 2nd block of the first process and the 2nd block of the third process. The third process passes the 1st block to the first process, passes the 2nd block to the second process, and receives the 3rd block of the first process and the 3rd block of the second process.

[0039] S12: After the communication, each i-th working process obtains the i-th block of all processes and performs a reduction operation on it. In the embodiment of the present application, the reduction operation is performed by taking the average value as an example.

[0040] In this embodiment, after completing the All-to-All communication, each process contains the momentum of other processes, that is, each i-th working process obtains the i-th block of all processes. At this time, the i-th process is subjected to a reduction operation. A reduction operation refers to an operation that combines multiple data values ​​into a single value. Common reduction operations include summing, averaging, maximum or minimum values, etc. In this embodiment of the application, the reduction operation selects the average value. Please continue to refer to Figure 1 , calculate the average value for each process, and then we can get the result after the reduction operation. At this point, the processes participating in the communication have completed the 1-bit Reduce-Scatter communication, which is called 1-bit Reduce-Scatter communication.

[0041] S13: After the average value is obtained, compression is performed and the second compression communication error is updated. Allgather is used to collect different blocks of each process to obtain the final aggregated momentum.

[0042] In this embodiment, after the reduction operation, the integrated data can be obtained as the local average. After obtaining the local average, the local average can be recompressed. Because 1-bit compression will lose all information except the sign bit. In this step, it is necessary to calculate the error caused by this compression and store it for use in subsequent iterations to compensate for the impact of compression. Then, each process uses Allgather to obtain the local average of other processes. Through Allgather, each process can obtain the global momentum. At this point, the processes participating in the communication complete the 1-bitAllreduce communication.

[0043] Based on the implementation of the above 1-bit Allreduce communication method, the embodiment of the present application provides a large model parallel training communication optimization method based on 1-bit compression. Exemplary, Figure 2 The flowchart of a large model parallel training communication optimization method based on 1-bit compression provided by an embodiment of the present application is shown. The method is applied to the ZeRO-1 stage (level) of ZeRO data. Figure 2 As shown, the large model parallel training communication optimization method may include the following steps:

[0044] S21: Execute forward calculation and reverse calculation of the model to determine gradient information.

[0045] In this embodiment, the entire model is trained by multiple GPUs, and on each GPU, there is at least one process for model training. In the embodiment of the present application, a GPU is provided on each computing device, and a process is used on each GPU as an example for description. The model and the data for training are divided into n parts, and the number of parts is the same as the total number of processes. For each process, a part of the entire model is used to perform training. During training, the forward calculation of the model is first performed to generate a predicted output. Then the reverse calculation of the model is performed to generate gradient information.

[0046] S22: Based on the gradient information, update the momentum in the local optimizer state.

[0047] In the ZeRO-1 stage of the original ZeRO data, after the gradient information is determined, Allreduce communication is performed immediately. In this embodiment, after the forward calculation and the backward calculation are completed, the gradient is no longer communicated immediately. In order to save memory and reduce data transmission, the storage space of the reused gradient is used to calculate and update the momentum partition of the current computing device with the corresponding partition in the gradient to obtain a new momentum to achieve local momentum update. At this time, the content of the gradient storage space is the updated momentum of the current computing device and the gradient required for other computing devices to update the momentum.

[0048] S23: Process the updated momentum via 1-bit Allreduce communication.

[0049] In this embodiment, by Figure 1 The 1-bit Allreduce shown communicates this new momentum tensor to achieve global momentum aggregation, and each computing device can obtain the complete aggregated momentum.

[0050] S24: Update the error generated during the compression process.

[0051] In this embodiment, since the original multi-precision floating-point momentum value is compressed to a 1-bit representation during the 1-bit compression process in 1-bit Allreduce, this compression method greatly reduces the accuracy of the data, and thus errors will occur. When compressing the momentum value, the system will record the error caused by this compression. In subsequent iterations, the recorded error is used to compensate for the compressed momentum value. In each iteration, the momentum value will be compressed again, and each compression may produce new errors. Therefore, it is necessary to continuously update the recorded error value so that more accurate compensation can be performed in the next iteration. In this way, although the data is compressed and decompressed in each iteration, by continuously compensating for the compression error, the impact of compression on the training process can be reduced, ensuring the accuracy and convergence of the training results.

[0052] S25: Update the model parameter partitions stored on each computing device using the aggregated momentum and the updated compression error.

[0053] In this embodiment, the complete aggregated momentum is obtained, and the momentum is used in combination with the variance to complete the parameter update of the model. At this time, there is no need to use Allgather to communicate the model parameters. After the parameters are updated, only the momentum of the partition corresponding to the current process is retained.

[0054] In this way, one round of model training is completed, and steps S21-S25 are iteratively executed in a loop until the model training is completed.

[0055] Figure 3 is a schematic diagram of the overall optimization process of ZeRO-1 provided in the embodiment of the present application, such as Figure 3 As shown in the figure, the model and data are divided into two parts for description. For each process, the model and data are processed. After forward calculation and reverse calculation, the gradient information is determined. Local momentum is updated and 1-bit Allreduce communication is performed. After completion, the compression error is updated. Based on the momentum information and variance information after 1-bit Allreduce communication, the model parameters are updated, thereby completing an iterative cycle.

[0056] Based on the optimization of ZeRO-1 of ZeRO data, the embodiment of the present application also provides a large model parallel training communication optimization method based on 1-bit compression, which is used for the optimization of ZeRO-2 of ZeRO data. Since ZeRO-2 further splits the gradient on the basis of ZeRO-1, ZeRO-2 is a method completed on the basis of ZeRO-1. Its steps are similar to Figure 2 The optimization part of ZeRO-1 shown is the same. For example, please refer to Figure 4 , after the forward and backward calculations are completed, the momentum is updated and stored in the corresponding partition of the gradient storage space. Next, 1-bitReduce-Scatter is used for communication (steps S11 and S12) to obtain the aggregated momentum. The aggregated momentum and fixed variance are used to update the model parameters. At this time, only part of the model parameters are updated on each computing device. Allgather is used to communicate the model parameters to obtain the complete model parameters.

[0057] Based on the optimization of ZeRO-2, the embodiment of the present application also provides a large model parallel training communication optimization method based on 1-bit compression, which is used for the optimization of ZeRO data ZeRO-3. Since ZeRO-3 is based on ZeRO-2 and also splits the model parameters, it is sufficient to use 1-bit Reduce-Scatter to replace the Reduce-Scatter in the original ZeRO-3 optimization to compress the gradient communication. For example, please refer to Figure 5 , Figure 5 Schematic diagram of the optimization process of ZeRO-3 provided in the embodiment of the present application. Figure 5 As shown, the optimized large model parallel training communication optimization method based on 1-bit compression applied to ZeRO-3 specifically includes: allocating the model and data to each process. Each process performs the forward calculation of the model. In this process, the Allgather communication operation is used to collect the complete model information on all computing devices to ensure that each device can perform the complete forward calculation. After completing the forward calculation, it is no longer necessary to store the complete model parameters on each computing device, so each process only retains the parameter partition responsible for the process, and discards the part that does not belong to the parameter partition to reduce storage space. Then the process performs the reverse calculation of the model to determine the gradient information. In this process, the Allgather communication operation is used to collect the gradient information on all computing devices to prepare for subsequent gradient compression and update. 1-bit Reduce-Scatter communication is performed and the compression error is updated. Then, the computing device discards the parameter information of non-partitions again and only retains the necessary data. Using the aggregated gradient information and variance, each process is calculated to independently update its assigned model parameter partition, thereby completing an iteration. After multiple iterations, until the model is trained.

[0058] The above is the communication optimization of large model parallel training based on 1-bit compression provided in the embodiment of the present application, including the communication optimization of ZeRO-1, ZeRO-2, and ZeRO-3 stages of ZeRO data. By adopting this method, the communication overhead of ZeRO data parallelization can be reduced and the efficiency of parallel training can be improved.

[0059] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any feature of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0060] Based on the method in the above embodiment, the embodiment of the present application also provides a large model parallel training communication optimization device based on 1-bit compression.

[0061] For example, Figure 6 A large model parallel training communication optimization device based on 1-bit compression is shown. Figure 6 As shown, the large model parallel training communication optimization device 600 based on 1-bit compression may include: a gradient information acquisition module 601, a momentum update module 602 and a processing module 603.

[0062] The gradient information acquisition module 601 is used to perform forward calculation and reverse calculation of the model to determine the gradient information.

[0063] Momentum update module 602, for updating the momentum in the local optimizer state based on the gradient information.

[0064] The processing module 603 is used to process the updated momentum through 1-bit Allreduce communication and update the error generated during the compression process.

[0065] The processing module 603 is further used to update the model parameter partition stored on each computing device using the aggregated momentum and variance.

[0066] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0067] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0068] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0069] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0070] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0071] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0072] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A large model parallel training communication optimization method, characterized in that: The method is applied to a system composed of multiple computing devices, each of which is configured with a GPU, and the GPU includes at least one process, and the process is used for a task partition in parallel training of a large model. The method is used to optimize the ZeRO-1 stage of ZeRO data, and the method includes: Perform forward and reverse calculations of the model to determine gradient information; Based on the gradient information, updating the momentum in the local optimizer state; The updated momentum is processed through 1-bit Allreduce communication, and the error generated during the compression process is updated; The aggregated momentum and variance are used to update the model parameter partitions stored on each computing device.

2. The method according to claim 1, characterized in that The 1-bit Allreduce communication includes: Send compressed momentum data blocks to other computing devices and receive compressed momentum data blocks from other computing devices using All-to-All communication primitives; Perform reduction operations on all received momentum data blocks; The Allgather communication primitive is used to collect momentum data blocks from all computing devices to complete the global aggregation of momentum.

3. The method according to claim 1, characterized in that The updating of the error generated during the compression process includes: recording the error generated during each compression and using it to compensate for the momentum after compression in the next iteration.

4. The method according to claim 1, characterized in that: The method is applied to the optimization of the ZeRO-2 stage of ZeRO data. After determining the gradient information, the method further includes: Use 1-bit Reduce-Scatter for communication and determine the momentum after aggregation; Based on the aggregated momentum and fixed variance, the model parameters are updated.

5. The method according to claim 4, characterized in that The method also includes: communicating model parameters via Allgather.

6. The method according to claim 1, characterized in that The method is applied to the optimization of the ZeRO-3 stage of ZeRO data, and the method comprises: Each process performs the forward calculation of the model and uses the All gather communication operation to collect the complete model information on all computing devices; After completing the forward calculation, each computing device discards the part that does not belong to the current parameter partition; Perform reverse calculation of the model, determine the gradient information, and use the Allgather communication operation to collect the gradient information on all computing devices; Perform 1-bit Reduce-Scatter communication and update compression error; The computing device again discards the parameter information of non-current partitions and only retains the necessary data of the current partition; Using the aggregated gradient information and variance, each process is computed to independently update its assigned partition of model parameters.

7. A large model parallel training communication optimization device, characterized in that: A system is deployed on a plurality of computing devices, each of which is configured with a GPU, wherein the GPU comprises at least one process, wherein the process is used for a task partition in parallel training of a large model, wherein the apparatus is used for optimizing the ZeRO-1 phase of ZeRO data, and the apparatus comprises: The gradient information acquisition module is used to perform forward and reverse calculations of the model and determine the gradient information; A momentum update module, configured to update the momentum in the local optimizer state based on the gradient information; A processing module, used to process the updated momentum through 1-bit Allreduce communication and update the error generated during the compression process; The processing module is also used to update the model parameter partition stored on each computing device using the aggregated momentum and variance.

Images