A large model training optimization method and device for domestic supercomputing systems

Abstract

A large-model training optimization method for domestic supercomputer systems is applied to computing devices in multiple domestic supercomputer systems. Each computing device is equipped with a GPU, and the GPU contains at least one process. The method is applied to the Megatron-DeepSpeed ​​framework and includes: determining the processes required for large-model training, and determining the process group to which each process belongs; based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, simultaneously constructing process groups, each of which includes at least one process; each process performs multiple forward and reverse calculations in the parallel training framework, and data exchange and synchronization are performed through the process group's communication mechanism. The forward and reverse calculations include collective communication. This method can improve the training efficiency of large-model training on domestic supercomputers.

Description

Technical Field

[0001] The present invention relates to the technical field of large-model parallel training, and in particular to a large-model training optimization method and device for domestic supercomputing systems. Background Art

[0002] In recent years, large language models based on Transformers have achieved tremendous success. Experience has shown that increasing the number of model parameters can effectively improve model performance. However, a large number of model parameters incurs significant computational and memory overhead. As the number of model parameters increases, effective model training becomes increasingly difficult. Improving training efficiency while increasing the number of parameters has become a pressing technical challenge. Summary of the Invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing equipment, computer storage medium and product containing computer program for large-scale model training optimization for domestic supercomputing systems, which can improve the training efficiency of large models.

[0004] In the first aspect, an embodiment of the present application provides a method for optimizing large-model training for domestic supercomputing systems, which is applied to computing devices of multiple domestic supercomputing systems, each computing device is configured with a GPU, and the GPU contains at least one process. The method is applied to the Megatron-DeepSpeed ​​framework, and the method includes: determining the processes required for large-model training, and determining the process group to which each process belongs; based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, constructing a process group at the same time, and the process group includes at least one process; each process performs multiple forward calculations and reverse calculations in the parallel training framework, and exchanges and synchronizes data through the communication mechanism of the process group, and the forward calculation and reverse calculation include collective communication.

[0005] In some possible implementations, in the Megatron-DeepSpeed ​​framework, inter-process communication is based on the message passing interface MPI.

[0006] In some possible implementations, the method further includes: using a half-precision floating-point type that complies with the IEEE-754 standard, and an FP16 library of arithmetic operators, type conversions, and mathematical functions, to add an FP16 data type to MPI.

[0007] In some possible implementations, during collective communication, MPI is used as the communication interface, UCX is used as the underlying communication framework, and UCC is used as the collective communication library.

[0008] In some possible implementations, during collective communication, a circular Reduce-Scatter algorithm is used to divide data into multiple blocks, and each process exchanges data blocks with its neighboring processes.

[0009] In some possible implementations, compressed data of all blocks are stored in the same sending buffer, and sufficient memory space is pre-allocated in front of the buffer to store the size of the compressed data.

[0010] In the second aspect, an embodiment of the present application provides an apparatus for optimizing large-model training for domestic supercomputing systems, which is deployed on computing devices of multiple domestic supercomputing systems. Each computing device is configured with a GPU, and the GPU contains at least one process. The large-model training adopts the Megatron-DeepSpeed ​​framework, and the apparatus includes: a determination module for determining the processes required for large-model training, and each process determines the process group to which it belongs; a construction module for simultaneously constructing a process group based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, and the process group includes at least one process; a training module for each process to perform multiple forward calculations and reverse calculations in the parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group. The forward calculation and reverse calculation include collective communication.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method as described in any one of the first aspects.

[0012] In a fourth aspect, an embodiment of the present application provides a computing device comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method as described in any one of the first aspects is executed.

[0013] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program, which, when the computer program product runs on a processor, enables the processor to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1This is a schematic diagram of a large model training optimization method for a domestic supercomputer system provided in an embodiment of the present application;

[0016] Figure 2 This is a flow chart of a large model training optimization method for a domestic supercomputer system provided in an embodiment of the present application;

[0017] Figure 3 This is a structural diagram of a large-model training and optimization device for a domestic supercomputer system provided in an embodiment of the present application;

[0018] Figure 4 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0021] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0022] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0024] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] Currently, most model training methods rely on accelerator cards for acceleration. However, due to the limitations of the computing power and storage capacity of a single accelerator card, distributed parallel training methods can be used for model training. Current distributed parallel training strategies for large models mainly include data parallelism, tensor parallelism, pipeline parallelism, and hybrid expert parallelism. Pipeline parallelism (PP) distributes different stages of the model to different devices, forming a processing pipeline. This is a type of model parallelism (MP). Its core concept is to partition the model. Data parallelism divides data into batches and processes them in parallel on multiple devices. Model parallelism distributes different parts of the model to different devices. Both pipeline parallelism and tensor parallelism partition the model itself, aiming to train larger models using the limited graphics memory of a single graphics card. Simply put, pipeline parallelism partitions the model horizontally, i.e., by layer; tensor parallelism partitions the model vertically. There are two main approaches to generating parallel training strategies: one leverages the experience of parallel system experts to manually set and optimize the parallel training strategy; the other uses tools or frameworks to generate the parallel training strategy. Although distributed parallel training methods can effectively support the training needs of large models, they also bring additional communication overhead.

[0026] Due to the lack of interconnectivity between accelerator cards in current domestic supercomputer systems, distributed parallel training of large models on these systems faces two challenges. First, because domestic accelerator cards have relatively limited storage space and weak computing power, more accelerator cards are required for computation. This results in a significant increase in training startup time as the number of accelerator cards increases. Second, due to the lack of interconnectivity between domestic accelerator cards, data transmission between accelerator cards must pass through the PCIE bus. As the number of accelerator cards increases, the communication volume increases, and the communication overhead seriously affects training efficiency.

[0027] In view of this, an embodiment of the present application provides a method for optimizing large model training for domestic supercomputing systems. Large model training adopts the Megatron-DeepSpeed ​​framework. Megatron-LM is a large and powerful Transformer model framework. Megatron-DeepSpeed ​​combines the zero redundancy optimizer (ZeRO) sharding and pipeline parallelism in the DeepSpeed ​​library with the tensor parallelism in Megatron-LM, so that large models can be trained in a very efficient way. Megatron-DeepSpeed ​​supports data parallelism, tensor parallelism, pipeline parallelism and zero redundancy optimizer. Among them, data parallelism means that the same initialized model is copied multiple times and each time it is fed a part of the minibatch. The processing is done in parallel, and all settings are synchronized at the end of each training step. Tensor parallelism means that each tensor is divided into multiple blocks. Instead of letting the entire tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. During the processing, each shard is processed in parallel on different GPUs, and the final results are synchronized at the end of the step. Tensor parallelism can also be called horizontal parallelism. Pipeline parallelism means that the model is split vertically (hierarchically) across multiple GPUs, so that only one or more layers of the model are placed on a single GPU. Each GPU processes different stages of the pipeline in parallel and handles a small portion of the batch. The execution of the zero-redundancy optimizer has similar tensor sharding as tensor parallelism, except that the entire tensor is rebuilt in time for forward or reverse calculations, so there is no need to modify the model. In this method, there are 3 parts in total. In the first part, there is a large model parallel training startup optimization method for domestic supercomputers; in the second part, there is a delay optimization method for collective communication; in the third part, there is a transmission optimization method for collective communication. In the embodiment of the present application, the domestic supercomputer is described by taking the "Eastern" supercomputer as an example, but the "Eastern" supercomputer does not constitute a limitation to this application.

[0028] Specifically, the first part includes the rapid construction of process groups and collective communication methods that support the FP16 data type. A process group is a collection of processes that can communicate with each other. For rapid process group construction, the tensor parallel, pipeline parallel, and data parallel process group initialization methods in the Megatron-DeepSpeed ​​framework have been modified. In the improved process group initialization method, each process first determines the process group to which it belongs; secondly, all processes participate in the construction of the process group simultaneously. For collective communication methods that support the FP16 data type, the FP16 data type is added to the message passing interface (MPI). Specifically, the FP16 data type is added to MPI using a half-precision floating-point type that complies with the IEEE-754 standard, as well as an FP16 library with corresponding arithmetic operators, type conversions, and common mathematical functions. FP16 refers to a data type that is encoded and stored using 2 bytes (16 bits).

[0029] In the second part, we will discuss latency optimization methods for collective communication. To address the high latency of MPI when executing GPU buffer communications, we will use Unified Collective Communication (UCC), a unified collective communication library that provides flexible, complete, and rich features for current and emerging programming models and runtimes. This approach integrates MPI and UCC into a communication backend that supports low-latency collective communication operations.

[0030] In the third part, a circular communication function called Reduce-Scatter is used, allowing each process to exchange data blocks with its neighbors. Reduce-scatter is a data manipulation and communication primitive in parallel computing, used to perform a global reduce operation on an input array followed by a scatter operation. The exchanged data blocks are obtained by partitioning the model activations according to the number of processes. Therefore, for large models, the data block size is also quite large due to the large amount of data. Typically, tensors such as activations and gradients that need to be communicated during large model training are located on the accelerator card (i.e., GPU). To mask the transmission and reception time overhead from compression and decompression, the original data is not compressed as a whole. Instead, the data is divided into small blocks for compressed communication before the Reduce-Scatter operation begins.

[0031] For example, Figure 1The embodiment of the present application provides a large model training optimization method for domestic supercomputer systems, which is applied to computing devices of multiple domestic supercomputer systems. Each computing device is configured with a GPU, and the GPU contains at least one process, which is used for a task partition in large model parallel training. The method is applied to the Megatron-DeepSpeed ​​framework, which is used for large model training. Figure 1 As shown, the method may include the following steps:

[0032] S11: Build process groups. The order of building process groups is based on the order of building in the Megatron-DeepSpeed ​​framework. Communication between process groups is based on MPI.

[0033] In this embodiment, the process group initialization methods for tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework can be modified. In parallel computing, a process group is a collection of processes that can communicate with each other, with each process responsible for executing a different part of the model or processing a different subset of data. Each process can be assigned to one or more GPUs to perform tasks. This assignment can be static (determined at program startup) or dynamic (assigned as needed at runtime). In tensor parallelism, pipeline parallelism, and data parallelism, processes within a process group need to work together and may be assigned to different GPUs to achieve parallel processing of different parts of the model or different subsets of data. For example, in data parallelism, each process may replicate the entire model and process different subsets of data on different GPUs. In tensor parallelism, different parts of the model may be distributed across different GPUs, with each process responsible for a portion of the model. In pipeline parallelism, different layers of the model may be executed on different GPUs, forming a processing pipeline. In this method, each process first determines the process group to which it belongs, allowing each process to identify its position in the parallel computing structure. Next, all processes participate in the construction of the process group simultaneously. This reduces initialization time and eliminates the need to wait for one process group to be fully constructed before starting the next process group. Since a process may belong to multiple parallel tasks simultaneously, that is, a process may belong to data parallel, tensor parallel, and pipeline parallel process groups at the same time, the original pipeline parallel, tensor parallel, and data parallel process group construction order in the Megatron-DeepSpeed ​​framework is maintained. The process group includes the first process group, the second process group, and the third process group. The first process group represents the pipeline parallel process group, the second process group represents the data parallel process group, and the third process group represents the tensor parallel process group.

[0034] Processes within a process group (which may be on different GPUs) need to communicate with each other to synchronize gradient updates, parameters, and other necessary information. Currently, the communication backend of the Megatron-DeepSpeed ​​framework is implemented using PyTorch, and the NCCL collective communication library is generally used, which does not have the relevant functions for process group construction. At the same time, since domestic supercomputing systems do not have the ability to directly interconnect between NVIDIA GPUs, they can only communicate between GPUs through PCI-E, which means that the NCCL library may not be suitable for domestic supercomputing systems. In order to meet the special needs of domestic supercomputing systems, an MPI version that supports GPU-Direct was selected as the communication backend. MPI provides process group-related functions that can be used to build process groups, which is not available in the NCCL library. By replacing the communication backend of PyTorch with MPI, the startup performance of parallel training of large models is optimized.

[0035] In some possible embodiments, an FP16 data type may be added to MPI.

[0036] In this embodiment, an IEEE-754-compliant half-precision floating-point type, along with corresponding arithmetic operators, type conversions, and an FP16 library of common mathematical functions, can be used to add an FP16 data type to MPI, thereby supporting mixed-precision training.

[0037] S12: Perform multiple forward calculations and reverse calculations to determine training results, wherein the forward calculations and reverse calculations include collective communication, and the collective communication includes UCC and unified communication X (UCX).

[0038] In this embodiment, both forward and backward computations require communication between processes. The sheer volume of communication leads to low training efficiency for large models. Therefore, this embodiment combines MPI with UCC and UCX to provide a high-performance communication backend for PyTorch. This approach optimizes communication latency.

[0039] Specifically, regarding the high latency problem of using MPI when performing GPU buffer communication, since UCC is a unified collective communication library, it provides flexible, complete, and rich features for current and emerging programming models and runtimes. It uses scalable and topology-aware algorithms, software implementation techniques, and in-network computing hardware acceleration engines to provide high-performance and scalable collective communication operations. UCC works in conjunction with UCX and can utilize the high-performance point-to-point communication operations and library utilities provided by UCX. For example, MPI can be used as a communication interface, UCX as the underlying communication framework, and UCC as a collective communication library. The MPI library can encapsulate the interfaces of UCC and UCX, so that MPI programs can directly use the collective communication operations provided by UCC while utilizing the optimized point-to-point communication capabilities of UCX. Set environment variables or use a specific launcher when starting the MPI program to use UCC / UCX as its communication backend.

[0040] In some possible embodiments, communication transmission may also be optimized.

[0041] Specifically, in parallel computing, especially scenarios involving large-scale data transmission and processing, data aggregation and dispersion between multiple computing nodes (such as GPUs) are often required. The Reduce-Scatter algorithm is a commonly used collective communication operation. In this embodiment, a circular communication function called Reduce-Scatter is used, where each process needs to exchange data blocks with its neighboring processes. In the parallel training framework Megatron-DeepSpeed, model data (such as activations and gradients) is divided into multiple blocks for distributed processing across multiple GPUs. Because the data blocks are derived by splitting the model activations according to the number of processes, the size of these data blocks becomes quite large for large models. Tensors such as activations and gradients that need to be communicated during large model training are typically located on accelerator cards, so accelerator cards are used to accelerate data compression. Compression and decompression on accelerator cards are much faster than on the CPU, so the time overhead of sending and receiving is greater than the time overhead associated with compression and decompression. To ensure that the sending and receiving time overhead is masked by the compression and decompression time overhead, the data is divided into small blocks for compression communication before the Reduce-Scatter process begins. However, since the storage space occupied by each compressed block of data is uncertain, correct decompression of the compressed data is impossible. Therefore, the compressed data for each block cannot be simply combined. To address this issue, the compressed data for all blocks can be stored in the same send buffer, with sufficient memory space (for example, 4 bytes per block) pre-allocated at the front of the buffer to store the compressed data size. By placing these blocks together (essentially an index) rather than storing them with the compressed data blocks, this design allows the algorithm to maintain a block starting position pointer based on the recorded compressed block size, indicating where the decompression operation should begin. This is more cache-friendly and therefore reduces overhead. During the decompression process at the receiving end, a block starting position pointer is maintained based on the recorded compressed block size, allowing the algorithm to know where to start the decompression operation for each block. This process is repeated until all data is aggregated and the Allreduce is complete. This optimization hides the time overhead of compression and decompression within the communication, further improving the performance of the ring-compressed Allreduce.

[0042] Figure 2 This is a flow chart of a large model training optimization process for a domestic supercomputer system provided by the embodiment of this application. Figure 2 As shown, after the method starts, the pipeline parallel process group, the data parallel process group, and the tensor parallel process group are quickly constructed. Figure 2The examples of starting to build a data parallel process group after the pipeline parallel process group is built, and starting to build a tensor parallel process group after the data parallel process group is built are used for demonstration, but do not constitute a limitation of this application. There is no sequential relationship between the construction of the pipeline parallel process group, the data parallel process group, and the tensor parallel process group. After the process group is built, each process starts forward calculation and reverse calculation. The forward calculation and reverse calculation processes both include collective communication between processes. During the collective communication process, communication optimization is performed by low-latency startup and low-traffic transmission.

[0043] The above is the method for optimizing large-model training for domestic supercomputing systems provided in the embodiment of the present application. By optimizing the startup during the large-model training process and optimizing the communication during the forward and reverse calculation processes, the efficiency of domestic supercomputing systems during large-model training can be improved.

[0044] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0045] Based on the method in the above embodiment, the embodiment of the present application also provides a device for large-model training optimization for domestic supercomputing systems.

[0046] For example, Figure 3 This paper shows an apparatus for optimizing large-model training for domestic supercomputing systems. The apparatus is deployed across multiple computing devices in these systems. Each computing device is equipped with a GPU, which contains at least one process. Large-model training utilizes the Megatron-DeepSpeed ​​framework. The apparatus 300 for optimizing large-model training for domestic supercomputing systems includes a determination module 301, a construction module 302, and a training module 303.

[0047] Among them, the determination module 301 is used to determine the processes required for large model training, and each process determines the process group to which it belongs.

[0048] A construction module 302 is used to simultaneously construct a process group based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, where the process group includes at least one process.

[0049] The training module 303 is used for each process to perform multiple forward calculations and reverse calculations in a parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group. The forward calculations and reverse calculations include collective communication.

[0050] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0051] The present application also provides a computing device 400. Figure 4 As shown, computing device 400 includes a bus 402, a processor 404, a memory 406, and a communication interface 408. Processor 404, memory 406, and communication interface 408 communicate with each other via bus 402. Computing device 400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 400.

[0052] The bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus 404 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 406, the processor 404, and the communication interface 408).

[0053] The processor 404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0054] The memory 406 may include volatile memory, such as random access memory (RAM). The processor 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0055] Memory 406 stores executable program code, which processor 404 executes to implement the functions of the aforementioned determination module 301, construction module 302, and training module 303, thereby implementing all or part of the steps of the method in the above-described embodiment. In other words, memory 406 stores instructions for executing all or part of the steps of the method in the above-described embodiment.

[0056] Alternatively, the memory 406 stores executable code, and the processor 404 executes the executable code to respectively implement the functions of the aforementioned large-model training and optimization device 300 for domestic supercomputer systems, thereby implementing all or part of the steps in the above-mentioned embodiment method. In other words, the memory 406 stores instructions for executing all or part of the steps in the above-mentioned embodiment method.

[0057] The communication interface 408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0058] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0059] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0060] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0061] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0062] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0063] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

Claims

1. A large model training optimization method for domestic supercomputer systems, characterized by: The method is applied to computing devices of multiple domestic supercomputer systems, each of which is configured with a GPU, and the GPU includes at least one process. The method is applied to the Megatron-DeepSpeed ​​framework, and the method includes: Determine the processes required for large model training, and determine the process group to which each process belongs; Based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, simultaneously construct the process group, wherein the process group includes at least one process; Each process performs multiple forward and reverse calculations in a parallel training framework, and performs data exchange and synchronization through the communication mechanism of the process group. The forward and reverse calculations include collective communication, which adopts a unified collective communication library UCC and a unified communication X framework UCX.

2. The method according to claim 1, characterized in that In the Megatron-DeepSpeed ​​framework, communication between processes is based on the message passing interface MPI.

3. The method according to claim 2, characterized in that The method further includes: using a half-precision floating-point type compliant with the IEEE-754 standard, and an FP16 library of arithmetic operators, type conversions, and mathematical functions to add an FP16 data type to the MPI.

4. The method according to claim 2, characterized in that During the collective communication, a circular Reduce-Scatter algorithm is used to divide the data into multiple blocks, and each process exchanges data blocks with its adjacent processes.

5. The method according to claim 4, characterized in that The compressed data of all blocks are stored in the same sending buffer, and sufficient memory space is pre-allocated in front of the buffer to store the size of the compressed data.

6. A large-model training optimization device for domestic supercomputer systems, deployed on computing devices of multiple domestic supercomputer systems, each computing device is configured with a GPU, and the GPU contains at least one process. The large-model training adopts the Megatron-DeepSpeed ​​framework, and the device includes: The determination module is used to determine the processes required for large model training and the process group to which each process belongs; A construction module is configured to simultaneously construct a process group based on the order of tensor parallelism, pipeline parallelism, and data parallelism in the Megatron-DeepSpeed ​​framework, wherein the process group includes at least one process; The training module is used for each process to perform multiple forward calculations and reverse calculations in a parallel training framework, and to exchange and synchronize data through the communication mechanism of the process group. The forward calculations and reverse calculations include collective communication, and the collective communication adopts the unified collective communication library UCC and the unified communication X framework UCX.

Images