Large model training optimization method and device for domestic supercomputing system

Abstract

A large model training optimization method oriented to domestic supercomputing systems is applied to computing devices of a plurality of domestic supercomputing systems, each computing device is provided with a GPU, each GPU comprises at least one process, the method is applied to a Megatron-DeepSpeed framework, and the method comprises the steps that processes needed by large model training are determined, and each process determines a process group to which the process belongs; based on the sequence of tensor parallelism, assembly line parallelism and data parallelism in the Megatron-DeepSpeed framework, a process group is constructed at the same time, and the process group comprises at least one process; each process executes forward calculation and reverse calculation for multiple times in a parallel training framework, data exchange and synchronization are carried out through a communication mechanism of a process group, and the forward calculation and the reverse calculation comprise set communication. According to the method, the training efficiency of large model training on domestic supercomputing can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large model parallel training, and in particular to a large model training optimization method and device for a domestic supercomputer system. Background Art

[0002] In recent years, large language models based on Transformer have achieved great success. Experience has shown that increasing the number of model parameters can effectively improve model performance. However, a large number of model parameters will generate a lot of computing and memory overhead. As the number of model parameters increases, it becomes increasingly difficult to effectively train the model. How to improve training efficiency while increasing the number of parameters has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing equipment, computer storage medium and product including a computer program for optimizing large model training for domestic supercomputer systems, which can improve the training efficiency of large models.

[0004] In the first aspect, an embodiment of the present application provides a method for optimizing large-model training for a domestic supercomputer system, which is applied to computing devices of multiple domestic supercomputer systems, each computing device is configured with a GPU, and the GPU includes at least one process. The method is applied to the Megatron-DeepSpeed ​​framework, and the method includes: determining the processes required for large-model training, and determining the process group to which each process belongs; based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, simultaneously constructing a process group, and the process group includes at least one process; each process performs multiple forward calculations and reverse calculations in the parallel training framework, and exchanges and synchronizes data through the communication mechanism of the process group, and the forward calculations and reverse calculations include collective communication.

[0005] In some possible implementations, in the Megatron-DeepSpeed ​​framework, communication between processes is based on the message passing interface MPI.

[0006] In some possible implementations, the method further includes: using a half-precision floating-point type that complies with the IEEE-754 standard, and an FP16 library of arithmetic operators, type conversions, and mathematical functions, to add an FP16 data type to MPI.

[0007] In some possible implementations, during collective communication, MPI is used as the communication interface, UCX is used as the underlying communication framework, and UCC is used as the collective communication library.

[0008] In some possible implementations, during collective communication, a circular Reduce-Scatter algorithm is used to divide data into multiple blocks, and each process exchanges data blocks with its neighboring processes.

[0009] In some possible implementations, compressed data of all blocks are stored in the same sending buffer, and sufficient memory space is pre-allocated in front of the buffer to store the size of the compressed data.

[0010] In the second aspect, an embodiment of the present application provides an apparatus for optimizing large-model training for domestic supercomputer systems, which is deployed on computing devices of multiple domestic supercomputer systems, each computing device is configured with a GPU, and the GPU contains at least one process. The large model training adopts the Megatron-DeepSpeed ​​framework, and the apparatus includes: a determination module, which is used to determine the processes required for large-model training, and each process determines the process group to which it belongs; a construction module, which is used to simultaneously construct process groups based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, and the process group includes at least one process; a training module, which is used for each process to perform multiple forward calculations and reverse calculations in the parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group, and the forward calculations and reverse calculations include collective communication.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in any one of the first aspects.

[0012] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method described in any one of the first aspects is executed.

[0013] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program. When the computer program product runs on a processor, the processor executes the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0015] Figure 1It is a schematic diagram of a large model training optimization method for a domestic supercomputer system provided in an embodiment of the present application;

[0016] Figure 2 It is a flowchart of a large model training optimization method for a domestic supercomputer system provided in an embodiment of the present application;

[0017] Figure 3 It is a structural schematic diagram of a large model training optimization device for a domestic supercomputer system provided in an embodiment of the present application;

[0018] Figure 4 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0021] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0022] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0023] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0024] To facilitate the understanding of the embodiments of the present application, the following will further explain and illustrate specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] At present, most model training methods rely on accelerator cards for acceleration. However, due to the limitations of the computing power and storage capacity of a single accelerator card, a distributed parallel training method can be used for model training. The current distributed parallel training strategies for large models mainly include data parallelism, tensor parallelism, pipeline parallelism, and hybrid expert parallelism. Among them, pipeline parallelism (PP) distributes different stages of the model to different devices to form a processing pipeline. It belongs to a type of model parallelism (MP). Its core idea is to split the model. Data parallelism divides the data into multiple batches and processes them in parallel on multiple devices. Model parallelism distributes different parts of the model to different devices. Both pipeline linear parallelism and tensor parallelism divide the model itself, with the aim of training larger models using limited single-card video memory. Simply put, pipeline parallelism divides the model horizontally, that is, divides the model by layer; tensor parallelism divides the model vertically. There are two main methods for generating parallel training strategies. One is to use the experience of parallel system experts to manually set and optimize the parallel training strategy, and the other is to use tools or frameworks to generate parallel training strategies. Although the distributed parallel training method can effectively support the training needs of large models, it also brings additional communication overhead.

[0026] Due to the lack of interconnection between accelerator cards in domestic supercomputer systems, there are two problems in performing large-model distributed parallel training on domestic supercomputer systems. One problem is that due to the relatively small storage space and weak computing power of domestic accelerator cards, more accelerator cards are required to participate in the calculation, resulting in a significant increase in training startup time as the number of accelerator cards increases. Another problem is that due to the lack of interconnection between domestic accelerator cards, data transmission between accelerator cards must pass through the PCIE bus. As the number of accelerator cards increases, the communication volume increases, and the communication overhead seriously affects the training efficiency.

[0027] In view of this, an embodiment of the present application provides a method for optimizing large model training for domestic supercomputer systems. The Megatron-DeepSpeed ​​framework is used for large model training. Megatron-LM is a large and powerful Transformer model framework. Megatron-DeepSpeed ​​combines the zero redundancy optimizer (ZeRO) sharding and pipeline parallelism in the DeepSpeed ​​library with the tensor parallelism in Megatron-LM, so that large models can be trained in a very efficient way. Megatron-DeepSpeed ​​supports data parallelism, tensor parallelism, pipeline parallelism, and zero redundancy optimizer. Among them, data parallelism means that the same initialization model is copied multiple times and is fed a part of a minibatch each time. The processing is done in parallel, and all settings are synchronized at the end of each training step. Tensor parallelism means that each tensor is divided into multiple blocks, not that the entire tensor resides on a single GPU, but that each shard of the tensor resides on its designated GPU. During the processing, each shard is processed in parallel on different GPUs, and the final result is synchronized at the end of the step. Tensor parallelism can also be called horizontal parallelism. Pipeline parallelism refers to the vertical (hierarchical) splitting of the model across multiple GPUs, so that only one or more layers of the model are placed on a single GPU. Each GPU processes different stages of the pipeline in parallel and handles a small portion of the batch. The execution of the zero-redundancy optimizer has similar tensor sharding as tensor parallelism, except that the entire tensor is rebuilt in time for forward or reverse calculations, so there is no need to modify the model. In this method, a total of 3 parts are included. In the first part, a large model parallel training startup optimization method for domestic supercomputers is included; in the second part, a delay optimization method for collective communication is included; in the third part, a transmission optimization method for collective communication is included. In the embodiment of the present application, the domestic supercomputer is described by taking the "Eastern" supercomputer as an example, but the "Eastern" supercomputer does not constitute a limitation of the present application.

[0028] Specifically, in the first part, the rapid construction of process groups and the collective communication method supporting FP16 data types are included. Among them, a process group refers to a set of processes that can communicate with each other. For the rapid construction of process groups, the tensor parallel, pipeline parallel, and data parallel process group initialization methods in the Megatron-DeepSpeed ​​framework are modified. In the improved process group initialization method, each process first determines the process group to which it belongs; secondly, all processes participate in the construction of the process group at the same time. For the collective communication method supporting FP16 data types, the FP16 data type is added to the message passing interface (MPI). Specifically, a half-precision floating point type that complies with the IEEE-754 standard, as well as the corresponding FP16 library of arithmetic operators, type conversions, and common mathematical functions are used to add the FP16 data type to MPI. FP16 means a data type that is encoded and stored in 2 bytes (16 bits).

[0029] In the second part, latency optimization methods for collective communication, for the high latency problem of MPI in executing GPU buffer communication, unified collective communication (UCC) is a unified collective communication library that provides flexible, complete, and rich features for current and emerging programming models and runtimes. MPI and UCC are integrated into a set of communication backends to support low-latency collective communication operations.

[0030] In the third part, the circular communication function Reduce-Scatter is used so that each process exchanges data blocks with its adjacent processes. Reduce-scatter is a data operation and communication primitive in parallel computing, which is used to perform a global reduce operation on the data of an input array and then perform a scatter operation. The exchanged data blocks are obtained by dividing the model activation according to the number of processes. Therefore, for large models, due to the large amount of data, the data block size is also quite large. Usually, the activation and gradient tensors that need to be communicated during the training of large models are located on the accelerator card (i.e., GPU). In order to achieve mutual masking of the time overhead of sending and receiving and compression and decompression, the original data is not compressed as a whole, but the data is divided into small blocks for compressed communication before the start of Reduce-Scatter.

[0031] For example, Figure 1The embodiment of the present application provides a large model training optimization method for a domestic supercomputer system, which is applied to computing devices of multiple domestic supercomputer systems, each of which is configured with a GPU, and the GPU includes at least one process, which is used for a task partition in large model parallel training. The method is applied to the Megatron-DeepSpeed ​​framework, which is used for large model training. Figure 1 As shown, the method may include the following steps:

[0032] S11: Build process groups, the process group building order is based on the building order in the Megatron-DeepSpeed ​​framework. Communication between process groups is based on MPI.

[0033] In this embodiment, the process group initialization method of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework can be modified. In parallel computing, a process group is a set of processes that can communicate with each other, and each process is responsible for executing different parts of the model or processing different data subsets. Each process can be assigned to one or more GPUs to perform tasks. This assignment can be static (determined at the beginning of the program) or dynamic (assigned as needed at runtime). In tensor parallelism, pipeline parallelism, and data parallelism, the processes in the process group need to work together, and they may be assigned to different GPUs to achieve parallel processing of different parts of the model or different data subsets. For example, in data parallelism, each process may copy the entire model and process different data subsets on different GPUs. In tensor parallelism, different parts of the model may be distributed on different GPUs, and each process is responsible for a part of the model. In pipeline parallelism, different layers of the model may be executed on different GPUs to form a processing pipeline. In this method, each process first determines the process group to which it belongs, so that each process can identify its position in the parallel computing structure. Then, all processes participate in the construction of the process group at the same time. This can reduce the initialization time, and there is no need to wait for a process group to be fully built before starting the construction of the next process group. Since a process may belong to multiple parallel tasks at the same time, that is, a process may belong to data parallel, tensor parallel and pipeline parallel process groups at the same time, the original pipeline parallel, tensor parallel and data parallel process group construction order in the Megatron-DeepSpeed ​​framework is maintained. The process group includes the first process group, the second process group and the third process group, where the first process group represents the pipeline parallel process group, the second process group represents the data parallel process group, and the third process group represents the tensor parallel process group.

[0034] Processes within a process group (which may be on different GPUs) need to communicate with each other to synchronize gradient updates, parameters, and other necessary information. Currently, the communication backend of the Megatron-DeepSpeed ​​framework is implemented through PyTorch, and the NCCL collective communication library is generally used, which does not have the relevant functions for process group construction. At the same time, since domestic supercomputer systems do not have the ability to directly interconnect between NVIDIA GPUs, they can only communicate through PCI-E between GPU-Direct, which means that the NCCL library may not be suitable for domestic supercomputer systems. In order to meet the special needs of domestic supercomputer systems, the MPI version that supports GPU-Direct was selected as the communication backend. MPI provides process group-related functions that can be used to build process groups, which is not available in the NCCL library. The communication backend of PyTorch is replaced with MPI, thereby optimizing the startup performance of large model parallel training.

[0035] In some possible embodiments, FP16 data types may also be added to MPI.

[0036] In this embodiment, a half-precision floating point type that complies with the IEEE-754 standard, as well as a corresponding FP16 library of arithmetic operators, type conversions, and common mathematical functions, can be used to add FP16 data types to MPI. In this way, mixed precision training can be supported.

[0037] S12: Perform multiple forward calculations and reverse calculations to determine training results, wherein the forward calculations and reverse calculations include collective communication, and the collective communication includes UCC and unified communication X (UCX).

[0038] In this embodiment, both forward and reverse calculations require communication between processes. Due to the huge amount of communication, the training efficiency of large models is low. Therefore, this embodiment combines MPI with UCC and UCX to provide a high-performance communication backend for PyTorch. In this way, communication delay optimization is achieved.

[0039] Specifically, for the high latency problem of using MPI to perform GPU buffer communication, since UCC is a unified collective communication library, it provides flexible, complete and rich features for current and emerging programming models and runtimes, and uses scalable and topology-aware algorithms, software implementation techniques and in-network computing hardware acceleration engines to provide high-performance and scalable collective communication operations. UCC works in conjunction with UCX and can utilize the high-performance point-to-point communication operations and library utilities provided by UCX. Exemplarily, MPI can be used as a communication interface, UCX as an underlying communication framework, and UCC as a collective communication library. The MPI library can encapsulate the interfaces of UCC and UCX, so that the MPI program can directly use the collective communication operations provided by UCC while utilizing the point-to-point communication capabilities optimized by UCX. Set environment variables or use a specific launcher when the MPI program starts to use UCC / UCX as its communication backend.

[0040] In some possible embodiments, communication transmission may also be optimized.

[0041] Specifically, in parallel computing, especially in scenarios involving large-scale data transmission and processing, it is often necessary to aggregate and disperse data between multiple computing nodes (such as GPUs), and the Reduce-Scatter algorithm is a commonly used collective communication operation. In this embodiment, a circular communication function Reduce-Scatter is used, and each process needs to exchange data blocks with its adjacent processes. In the parallel training framework Megatron-DeepSpeed, the data of the model (such as activation and gradient) will be divided into multiple blocks for distributed processing on multiple GPUs. Since the data blocks are obtained by dividing the model activation according to the number of processes, the size of these data blocks becomes quite large for large models. Tensors such as activations and gradients that need to be communicated during the training of large models are usually located on accelerator cards, so accelerator cards are used to accelerate data compression. Compression and decompression on accelerator cards are much faster than compression and decompression on CPUs, so the time overhead of sending and receiving is greater than the time overhead associated with compression and decompression. In order to achieve mutual masking of the time overhead of sending and receiving and compression and decompression, the data is divided into small blocks for compressed communication before Reduce-Scatter starts. However, since the storage space occupied by each block of data after compression is uncertain, the compressed data cannot be decompressed correctly. Therefore, the compressed data of each block cannot be simply combined together. To solve this problem, the compressed data of all blocks can be stored in the same sending buffer, and sufficient memory space (for example, 4 bytes per block) is pre-allocated in front of the buffer to store the size of the compressed data. Putting these blocks together (essentially an index) instead of storing them with the compressed data blocks allows the algorithm to maintain a block start position pointer based on the recorded compressed block size to tell where the decompression operation should start, which is more cache-friendly and therefore has lower overhead. During the decompression process at the receiving end, a block start position pointer is maintained based on the recorded compressed block size so that the algorithm knows where the decompression operation of each block should start. Repeat this process until all data aggregation Allreduce is completed. Through this optimization, the time overhead of compression and decompression can be hidden inside the communication, which further improves the performance of ring compression Allreduce.

[0042] Figure 2 This is a flow chart of a large model training optimization process for a domestic supercomputer system provided by an embodiment of the present application. Figure 2 As shown, after the method starts, a pipeline parallel process group, a data parallel process group, and a tensor parallel process group are quickly constructed. Figure 2The example of starting to build a data parallel process group after the pipeline parallel process group is built, and starting to build a tensor parallel process group after the data parallel process group is built is used for demonstration, but it does not constitute a limitation of the present application. There is no sequential relationship between the construction of the pipeline parallel process group, the data parallel process group, and the tensor parallel process group. After the process group is built, each process starts forward calculation and reverse calculation. In the forward calculation and reverse calculation process, collective communication between processes is included. In the process of collective communication, communication optimization is performed in a low-latency startup mode and a low-communication volume transmission mode.

[0043] The above is the method for optimizing large model training for domestic supercomputer systems provided in the embodiment of the present application. By optimizing the startup during the large model training process and optimizing the communication during the forward and reverse calculation processes, the efficiency of domestic supercomputer systems during large model training can be improved.

[0044] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any feature of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0045] Based on the method in the above embodiment, the embodiment of the present application also provides a device for large model training optimization for domestic supercomputer systems.

[0046] For example, Figure 3 A device for optimizing large model training for domestic supercomputer systems is shown, which is deployed on computing devices of multiple domestic supercomputer systems, each of which is equipped with a GPU, and the GPU includes at least one process. The large model training adopts the Megatron-DeepSpeed ​​framework. The device 300 for optimizing large model training for domestic supercomputer systems includes: a determination module 301, a construction module 302, and a training module 303.

[0047] Among them, the determination module 301 is used to determine the processes required for large model training, and each process determines the process group to which it belongs.

[0048] The construction module 302 is used to simultaneously construct a process group based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, and the process group includes at least one process.

[0049] The training module 303 is used for each process to perform multiple forward calculations and reverse calculations in a parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group. The forward calculations and reverse calculations include collective communication.

[0050] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0051] The present application also provides a computing device 400. Figure 4 As shown, computing device 400 includes: bus 402, processor 404, memory 406 and communication interface 408. Processor 404, memory 406 and communication interface 408 communicate through bus 402. Computing device 400 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 400.

[0052] The bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus 404 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 404 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 406, the processor 404, and the communication interface 408).

[0053] The processor 404 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0054] The memory 406 may include a volatile memory, such as a random access memory (RAM). The processor 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0055] The memory 406 stores executable program codes, and the processor 404 executes the executable program codes to respectively implement the functions of the aforementioned determination module 301, construction module 302, and training module 303, thereby implementing all or part of the steps of the method in the above embodiment. That is, the memory 406 stores instructions for executing all or part of the steps in the method in the above embodiment.

[0056] Alternatively, the memory 406 stores executable codes, and the processor 404 executes the executable codes to respectively implement the functions of the aforementioned large model training optimization device 300 for domestic supercomputer systems, thereby implementing all or part of the steps in the above-mentioned embodiment method. That is, the memory 406 stores instructions for executing all or part of the steps in the above-mentioned embodiment method.

[0057] The communication interface 408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0058] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0059] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0060] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0061] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0062] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0063] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A large model training optimization method for domestic supercomputer systems, characterized in that: The computing devices applied to multiple domestic supercomputer systems, each computing device is configured with a GPU, and the GPU includes at least one process. The method is applied to the Megatron-DeepSpeed ​​framework, and the method includes: Determine the processes required for large model training, and determine the process group to which each process belongs; Based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, the process group is constructed simultaneously, and the process group includes at least one process; Each of the processes executes multiple forward calculations and reverse calculations in a parallel training framework, and performs data exchange and synchronization through the communication mechanism of the process group, wherein the forward calculations and reverse calculations include collective communication.

2. The method according to claim 1, characterized in that In the Megatron-DeepSpeed ​​framework, communication between processes is based on the message passing interface MPI.

3. The method according to claim 2, characterized in that The method further includes: using a half-precision floating point type that complies with the IEEE-754 standard, and an FP16 library of arithmetic operators, type conversions, and mathematical functions to add an FP16 data type to the MPI.

4. The method according to claim 2, characterized in that: During the collective communication, MPI is used as the communication interface, UCX is used as the underlying communication framework, and UCC is used as the collective communication library.

5. The method according to claim 2, characterized in that: During the collective communication, a circular Reduce-Scatter algorithm is used to divide the data into a plurality of blocks, and each of the processes exchanges data blocks with its adjacent processes.

6. The method according to claim 5, characterized in that The compressed data of all blocks are stored in the same sending buffer, and sufficient memory space is pre-allocated in front of the buffer to store the size of the compressed data.

7. A large model training optimization device for domestic supercomputer systems, deployed on computing devices of multiple domestic supercomputer systems, each computing device is configured with a GPU, and the GPU includes at least one process. The large model training adopts the Megatron-DeepSpeed ​​framework, and the device includes: The determination module is used to determine the processes required for large model training and the process group to which each process belongs; A construction module, configured to simultaneously construct a process group based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, wherein the process group includes at least one process; The training module is used for each of the processes to perform multiple forward calculations and reverse calculations in a parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group, wherein the forward calculations and reverse calculations include collective communication.

Images