Model training time optimization method and electronic device

Abstract

The application discloses a model training time optimization method and an electronic device, and relates to the technical field of computers. According to the global identifier and parallel dimension information of the training process, the target group identifier of the training process is determined, and then according to the target group identifiers of the training processes, the multiple training processes are divided into a corresponding number of local process groups. The process of determining the target group identifier of each training process does not interfere with each other, the multiple training processes can be parallelly determined to form local process groups, the division time of the local process groups is shortened, the training start time is shortened, and in the communication process of the accelerator, the compression process of the to-be-communicated data block and the communication process of the to-be-communicated compressed file overlap in time, the communication efficiency between the accelerators is improved, and the overall time consumption of the model training is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for optimizing model training time and an electronic device. Background Technology

[0002] Taking a pre-trained language model based on the Transformer architecture as an example, the model's performance improves with the increase in the number of model parameters. However, the increase in the number of model parameters also leads to an increase in computational, memory, and communication overhead. Therefore, a distributed cluster training approach is currently commonly used, training the pre-trained language model on multiple nodes, with each node equipped with an accelerator card to improve computing power.

[0003] However, as the number of nodes and the number of accelerator cards in each node increases, although the computing power and other resources for model training are improved, due to the characteristics of multi-node model training and the application of accelerator cards, there are problems such as excessively long training startup time and low communication efficiency between accelerator cards, which in turn increases the overall time consumption of model training. Summary of the Invention

[0004] This application provides a method and electronic device for optimizing model training time, which at least solves the problems of excessively long training start-up time and low communication efficiency between accelerator cards in related technologies, which increase the overall time consumption of model training.

[0005] This application provides a method for optimizing model training time, including:

[0006] Obtain the global identifiers and parallel dimension information of multiple training processes of the model to be trained;

[0007] For any of the training processes, the target group identifier of the training process is determined based on the global identifier and parallel dimension information of the training process.

[0008] Based on the target group identifier of each training process, the plurality of training processes are divided into a corresponding number of local process groups;

[0009] Each of the local process groups is assigned to a corresponding node so that the training process in the local process group can be run on the node.

[0010] Obtain the communication data generated by the first accelerator card in the node in response to the training process;

[0011] The data to be communicated is divided into multiple data blocks to be communicated, and each individual data block to be communicated is compressed to obtain a compressed file to be communicated.

[0012] During the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card, the uncompressed data block to be communicated is compressed so that the compression process of the data block to be communicated and the communication process of the compressed file to be communicated overlap in time.

[0013] This application also provides a model training time optimization device, including:

[0014] The first acquisition module is used to acquire the global identifiers and parallel dimension information of multiple training processes of the model to be trained;

[0015] The determination module is used to determine the target group identifier of any training process based on the global identifier and parallel dimension information of the training process.

[0016] The partitioning module is used to divide the plurality of training processes into a corresponding number of local process groups based on the target group identifier of each training process.

[0017] The training module is used to assign each of the local process groups to a corresponding node so as to run the training process in the local process group based on the node;

[0018] The second acquisition module is used to acquire communication data generated by the first accelerator card in the node in response to the training process.

[0019] The compression module is used to divide the data to be communicated into multiple data blocks to be communicated, and to compress each individual data block to be communicated at a time to obtain a compressed file to be communicated.

[0020] The communication module is used to compress uncompressed data blocks to be communicated during the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card, so that the compression process of the data blocks to be communicated and the communication process of the compressed file to be communicated overlap in time.

[0021] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model training time optimization methods.

[0022] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described model training time optimization methods.

[0023] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model training time optimization methods.

[0024] By means of this application, since the target group identifier of the training process is determined based on the global identifier and parallel dimension information of the training process, and then the multiple training processes are divided into a corresponding number of local process groups based on the target group identifier of each training process, the process of determining the target group identifier of each training process does not interfere with each other, and multiple training processes can determine local process groups in parallel, which shortens the division time of local process groups and thus shortens the training start time. Furthermore, in the communication process of the accelerator card, the compression process of the data block to be communicated and the communication process of the compressed file to be communicated have a certain time overlap, which improves the communication efficiency between accelerator cards and thus reduces the overall time consumption of model training. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the structure of the model training time optimization system based on the embodiments of this application;

[0027] Figure 2 A flowchart illustrating the model training time optimization method provided in this application embodiment;

[0028] Figure 3 A schematic diagram of the structure of an exemplary local process group provided in an embodiment of this application;

[0029] Figure 4 This is a timing diagram of the accelerator card communication provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of the model training time optimization device provided in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0033] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0034] In recent years, large language models (preprocessed language models) based on Transformers have achieved great success. Experience has shown that increasing the number of model parameters can effectively improve model performance. However, a large number of model parameters generates significant computational, memory, and communication overhead. As the number of model parameters increases, effectively training the model becomes increasingly difficult.

[0035] Currently, most model training methods rely on accelerator cards for acceleration. However, due to the limitations of the computing power and storage capacity of a single accelerator card, distributed parallel training methods are the mainstream approach to improve model training efficiency by utilizing multiple accelerator cards, including data parallelism, tensor parallelism, and pipelined parallelism. Data parallelism divides the training data into batches, with each accelerator card responsible for computing only a portion of the data. This method requires each accelerator card to store a copy of the entire model, and as the number of model parameters increases, a single accelerator card may not be able to meet the model storage requirements. Tensor parallelism and pipelined parallelism divide the entire model into sub-models, with each accelerator card storing only a subset of the model. When training large models, it is often necessary to combine multiple distributed parallel training methods.

[0036] While distributed parallel training methods can effectively support the training needs of large models, they also introduce additional communication overhead. In data-parallel training methods, each accelerator card has a copy of the model and a subset of the input data. During backward computation, the gradients of all accelerator cards need to be synchronized before updating the model (i.e., the Allreduce operation), and the amount of communication data is the same as the amount of model parameters. In pipelining parallel training methods, the model is divided into blocks, requiring point-to-point communication to pass activation values ​​(i.e., Send and Recv operations). In tensor parallel training methods, each accelerator card possesses a portion of the model parameters, and activation values ​​need to be synchronized during forward and backward computation through reduction operations (i.e., the Allreduce operation).

[0037] Due to the lack of interconnection between accelerator cards in the current distributed cluster, large-scale distributed parallel training of models on the distributed cluster faces two challenges: (1) Due to the relatively small storage space and weak computing power of the accelerator cards, more accelerator cards are needed to participate in the computation, which leads to a significant increase in training startup time as the number of accelerator cards increases; (2) Due to the lack of interconnection between accelerator cards, data transmission between accelerator cards must go through the PCI-E bus. As the number of accelerator cards increases, the communication volume increases, and the communication overhead seriously affects the training efficiency.

[0038] To address the aforementioned technical problems, embodiments of this application provide a model training time optimization method and electronic device. The method includes: acquiring global identifiers and parallel dimension information of multiple training processes of a model to be trained; for any training process, determining a target group identifier based on the global identifier and parallel dimension information of the training process; dividing the multiple training processes into a corresponding number of local process groups based on the target group identifiers of each training process; allocating each local process group to a corresponding node to run the training processes in the local process group based on the node; acquiring communication data generated by a first accelerator card in the node in response to the training process; dividing the communication data into multiple communication data blocks, compressing a single communication data block each time to obtain a communication compressed file; during the process of the first accelerator card sending the communication compressed file to a second accelerator card, compressing the uncompressed communication data blocks so that the compression process of the communication data blocks and the communication process of the communication compressed file overlap in time. The method provided by the above scheme determines the target group identifier of the training process based on the global identifier and parallel dimension information of the training process. Then, based on the target group identifier of each training process, multiple training processes are divided into a corresponding number of local process groups. The process of determining the target group identifier of each training process does not interfere with each other. This allows multiple training processes to determine local process groups in parallel, shortening the division time of local process groups and thus shortening the training startup time. Furthermore, during the communication process of the accelerator cards, the compression process of the data block to be communicated and the communication process of the compressed file to be communicated have a certain time overlap, which improves the communication efficiency between accelerator cards and thus reduces the overall time consumption of model training.

[0039] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] The specific application environment architecture or specific hardware architecture on which the optimization method for model training time depends is described here.

[0041] First, the structure of the model training time optimization system on which this application is based will be described:

[0042] The model training time optimization method and electronic device provided in this application are applicable to shortening the overall training time of models in distributed cluster training scenarios. Figure 1 The diagram shown is a structural schematic of the model training time optimization system based on the embodiments of this application. It mainly includes a distributed cluster and a model training time optimization device. The model training time optimization device can be any node in the distributed cluster. The distributed cluster includes multiple nodes, and each node is equipped with multiple accelerator cards. Each node in the distributed cluster runs the model training process based on the accelerator cards. The model training time optimization device shortens the overall time of model training in the distributed cluster by dividing local process groups and optimizing communication between accelerator cards.

[0043] This application provides a method for optimizing model training time, used to shorten the overall training time in distributed cluster training scenarios. The execution subject of this application is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, and other electronic devices that can be used to optimize model training time.

[0044] like Figure 2 The diagram shown is a flowchart illustrating the model training time optimization method provided in this application embodiment. The method includes:

[0045] Step 201: Obtain the global identifiers and parallel dimension information of multiple training processes of the model to be trained.

[0046] It should be noted that the accelerator cards in the node complete the corresponding model training tasks by executing training processes. The global identifier of the training process, also known as the rank, is a globally unique number assigned to each training process. The parallel dimensions of the training process are at least divided into data parallelism, tensor parallelism, and pipeline parallelism. The parallel dimension information includes the configuration parameters of various parallel dimensions.

[0047] Step 202: For any training process, determine the target group identifier of the training process based on the global identifier and parallel dimension information of the training process.

[0048] Specifically, the target group identifier can be determined independently for each model process, meaning that the target group identifier can be determined in parallel during the training of multiple models. The target group identifier is used to characterize the communication group (local process group) to which the process belongs. Communication groups are at least divided into three types: data parallel groups, tensor parallel groups, and pipeline parallel groups, with multiple instances of each type.

[0049] Step 203: Divide multiple training processes into a corresponding number of local process groups according to the target group identifier of each training process.

[0050] Specifically, after determining the target group identifier for each training process, the division of all local process groups can be completed by calling collective communication operations (such as Comm_split).

[0051] Local process groups, also known as communication domains, are collections of training processes that need to communicate frequently with each other.

[0052] Step 204: Assign each local process group to the corresponding node to run the training process in the local process group based on the node.

[0053] Specifically, after the local process group is divided, it is allocated to the computing nodes by the cluster scheduling system to start execution and enter the model training phase.

[0054] It should be noted that large-scale parallel model training involves multiple parallel training methods, including data parallelism, tensor parallelism, and pipelined parallelism. Since both data parallelism and tensor parallelism involve local aggregate communication between accelerator cards, the model training startup process involves the construction of local process groups for different parallel training methods to facilitate local aggregate communication between processes during model training. Analysis of the parallel training startup process reveals that the most time-consuming part is the process group initialization. The construction of thousands of process groups accounts for approximately 80% of the total model training initialization time. Currently, during process group construction, all processes are essentially executed serially, resulting in significant loop overhead and severe process idle waiting. This application's embodiment optimizes the local process group partitioning process (initialization process), significantly shortening the model training initialization time, i.e., reducing the model training startup phase time.

[0055] Step 205: Obtain the communication data generated by the first accelerator card in the node in response to the training process.

[0056] The data to be communicated can be communication data transmitted between different accelerator cards within a node to achieve collaborative training.

[0057] Step 206: Divide the data to be communicated into multiple data blocks to be communicated, and compress each individual data block to be communicated at a time to obtain a compressed file to be communicated.

[0058] Specifically, by dividing the data to be communicated into multiple data blocks and compressing each individual data block at a time, the data block compression task can be interspersed during the communication process, avoiding long communication wait times caused by compressing the entire large data block (data to be communicated) at once.

[0059] Step 207: During the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card, the uncompressed data block to be communicated is compressed so that the compression process of the data block to be communicated and the communication process of the compressed file to be communicated overlap in time.

[0060] For example, the Nth data block is first compressed to obtain the Nth compressed file. Then, the sending operation (communication) of the Nth compressed file is initiated. While sending the Nth compressed file, the compression of the N+1th data block begins, and so on. The compression operation and the communication operation can overlap in time, thereby reducing the total time from the start of data acquisition to the completion of data communication, thus improving the communication efficiency between accelerator cards and reducing the overall time consumption of model training.

[0061] Based on the above embodiments, as an implementable approach, in one embodiment, for any training process, the target group identifier of the training process is determined according to the global identifier and parallel dimension information of the training process, including:

[0062] Step 2021: Determine the position of the training process in the pipeline parallel group based on the global identifier, data parallelism, and tensor parallelism of the training process.

[0063] Step 2022: Determine the data parallel group identifier of the training process based on its position in the pipeline parallel group, global identifier, data parallelism, and tensor parallelism.

[0064] The parallel dimension information includes data parallelism and tensor parallelism, and the target group identifier includes the data parallelism group identifier. In this embodiment, since each process may belong to data parallelism, tensor parallelism, and pipelined parallelism process groups simultaneously, the original process group construction order of the three stages of pipelined parallelism, tensor parallelism, and data parallelism is maintained.

[0065] Specifically, in one embodiment, the position of the training process within the pipelined parallel group can be determined based on the following formula:

[0066]

[0067] in, This indicates the position of the training process within the pipeline's parallel group. Indicates a global identifier. Indicates the amount of data parallelism. Represents a parallel tensor.

[0068] Accordingly, in one embodiment, the data parallel group identifier of the training process can be determined based on the following formula:

[0069]

[0070] in, The data parallel group identifier represents the training process. Indicates a global identifier. Represents tensor parallel quantities. This indicates the position of the training process within the pipeline's parallel group. This represents the offset of the training process in the pipeline parallel dimension.

[0071] For example, the initialization process of a local process group provided in this application embodiment is as follows:

[0072] pp_local_rank = rank / (dp_size×tp_size)

[0073] group_id = rank%tp_size + pp_local_rank×tp_size

[0074] dp_group = comm_split(group_id)

[0075] Wherein, pp_local_rank is rank is dp_size is group_id is `dp_group` refers to the local process group, and `comm_split` indicates the MPI version supporting GPU-Direct, providing process group-related functions. The pseudocode for the initialization implementation described above illustrates how, in a distributed training environment, processes are allocated to different communication groups based on the global rank (global identifier) ​​and the size of different parallel dimensions. This method effectively organizes and manages computational resources in large-scale distributed training, improving training efficiency.

[0076] Specifically, in one embodiment, after dividing multiple training processes into a corresponding number of local process groups according to the target group identifier of each training process, for any local process group, the training processes can be sorted in the local process group according to the global identifier of each training process.

[0077] For example, such as Figure 3The diagram shown illustrates an exemplary local process group structure provided in this application embodiment. The numbers within the small rectangles represent global identifiers. Processes within a large rectangle containing four small rectangles are processes within the same tensor group (tp_size=4). Two horizontal tensor groups are within the same data parallel group (dp_size=2), and two vertical tensor groups are within the same pipeline parallel group (corresponding to pp_local_rank of 0 and 1, respectively). During local process group initialization, the pipeline number (pp_local_rank) for each process is first calculated. Then, based on the process's position within the tensor group and pipeline group, the data parallel process group number (group_id) to which the process belongs is calculated. Finally, data parallel process groups are divided according to the group_id. These data parallel process groups are used for data transmission during model data parallel training.

[0078] Specifically, in one embodiment, local process groups assigned to different nodes communicate with each other via a hierarchical hybrid communication backend.

[0079] The layered hybrid communication backend includes a strategy optimization layer, a transmission optimization layer, and a basic communication layer. The strategy optimization layer is used to select the optimal communication strategy, the transmission optimization layer is used to select the optimal communication channel, and the basic communication layer is used to perform data communication based on the optimal communication strategy and the optimal communication channel.

[0080] It should be noted that the current local process group uses MPI as the communication backend for communication to accelerate the card buffer, which has a high communication latency. For example, when training the LLaMA(7B) model, as the number of communication bytes increases, the communication latency increases significantly at 128k, and continues to increase with the number of communication bytes.

[0081] To address this technical problem, this application provides a layered hybrid communication backend. For the high latency issue of MPI in executing GPU (accelerator card) buffer communication, the Policy Optimization Layer (UCC) provides a unified set of communication libraries, offering flexible, complete, and rich features for current and emerging programming models and runtimes. UCC utilizes scalable and topology-aware algorithms, software implementation techniques, and network-in-network computing hardware acceleration engines to provide high-performance and scalable set of communication operations. This application combines MPI (Basic Communication Layer) with UCC and UCX (Transmission Optimization Layer) to provide a high-performance communication backend for PyTorch. Since the startup method for model training with different parameter sizes is the same, the method proposed in this application is applicable to model training with any parameter size.

[0082] Specifically, in one embodiment, the strategy optimization layer can proactively probe the cluster's hardware topology during initialization to identify which accelerator cards reside within the same node. Accelerator cards within the same node can be interconnected via NVLink high-speed interconnection. The target communication strategy includes the selection of the communication network; within the same node, the NVLink high-speed interconnection communication network is selected for communication. Furthermore, the communication strategy also includes the selection of the communication algorithm. The strategy optimization layer has a built-in performance model that predicts the performance of different communication algorithms (such as Ring, Tree, Halving-Doubling, etc.) under the current data transmission volume. Specifically, if the data transmission volume is low, a tree-based communication algorithm can be selected to reduce latency; for high data transmission volumes, the Ring algorithm is selected to better utilize bandwidth. The strategy optimization layer automatically selects the optimal solution based on the runtime environment, thereby reducing communication latency.

[0083] Based on the above embodiments, as one implementable approach, in one embodiment, the data to be communicated is divided into multiple data blocks to be communicated, including:

[0084] Step 2061: Determine the target size of the data block to be communicated based on the preset accelerator card cache space size;

[0085] Step 2062: Divide the data to be communicated into multiple data blocks according to the target size of the data block to be communicated.

[0086] Specifically, by analyzing the time of various operators and communication operations in the Transformer layer during the forward computation process of large model pre-training, the All-Reduce communication time brought about by tensor parallelism accounted for 25.4%, and the Send and Recv communication time brought about by pipeline parallelism accounted for 18.51%, for a total communication time of 43.91%. During data communication, computational resources were idle, resulting in resource underutilization. In tensor parallelism, for a single Transformer layer, both forward and backward computations require All-reduce communication after completing the linear matrix multiplication (GEMM) of the attention block and multilayer perceptron block and before Dropout. That is, a single Transformer layer requires two All-reduce communications during the forward computation and two during the backward computation. The communication content is the calculated activations, with a size of sbh (s is the sequence length, b is the batch size, and h is the hidden layer size), and the data type is FP16. Therefore, in one iteration, a Transformer layer performs a total of 4 All-reduce communications, generating 16sbh bytes of communication data.

[0087] In related technologies, for communication primitives like Reduce that involve set-compact reduction, data from all processes within the same communication domain (local process group) needs to be reduced and computed with data from other processes. Unlike communication primitives such as Allgather, Bcast, and All-to-All, data transmitted in this communication mode participates in reduction computation and is then updated. Therefore, data transmission and compression / decompression timing similar to the Allgather algorithm cannot be used. This application's embodiment employs a Reduce-Scatter ring structure to adjust the aforementioned timing, where each process needs to exchange data blocks with its neighboring processes. These data blocks are obtained by dividing the activation values ​​of a large model according to the number of processes, resulting in a considerable size for large models. Typically, the activation and gradient tensors requiring communication during large model training reside on an accelerator card. Therefore, this application's embodiment uses an accelerator card to accelerate the SZx compression algorithm. Performing compression and decompression on an accelerator card is much faster than performing it on a non-accelerator card; therefore, the time overhead of sending and receiving is greater than the time overhead associated with compression and decompression. In the CPR-P2P algorithm, compression and decompression occur before and after all communications, respectively. Therefore, one round (Reduce-Scatter) generates three types of overhead: compression and decompression of data blocks, sending and receiving compressed data blocks, and reduction of a data block. These three types of operations need to be executed sequentially, which takes a long time.

[0088] In order to solve this technical problem, this application embodiment first determines the target size of the data block to be communicated based on the preset accelerator card cache space size; then, according to the target size of the data block to be communicated, the data to be communicated is divided into multiple data blocks to be communicated, each data block to be communicated can be compressed and decompressed independently, and its compression and decompression process can be performed simultaneously with the communication operation process.

[0089] The preset accelerator card cache space size is the size of the accelerator card's transmit buffer, such as 4 bytes, used to store compressed data (compressed files).

[0090] Specifically, to achieve mutual masking of the time overhead of sending, receiving, compression, and decompression, this embodiment does not compress the original data (data to be communicated) as a whole. Instead, it divides the data into small blocks for compression and communication before the Reduce-Scatter process begins. Since the storage space occupied by each compressed data block is uncertain, which could lead to incorrect decompression, the compressed data of each block cannot be simply combined together. To solve this problem, this embodiment stores the compressed data (compressed files) of all blocks in the same sending buffer, and pre-allocates sufficient memory space (4 bytes per compressed file) in front of the buffer to store the size of the compressed data. Because the size of each compressed file is fixed (4 bytes), putting these cached files together is essentially an index, rather than storing the size information of the compressed files together. This design is more cache-friendly and therefore has lower overhead. During decompression, a block start position pointer is maintained based on the recorded compressed file size to inform the algorithm where the decompression operation for each block should begin. This process is repeated until all data is aggregated.

[0091] Furthermore, in one embodiment, a single block of data to be communicated can also be compressed during the process of the first accelerator card receiving an external compressed file sent by any accelerator card.

[0092] Among them, the accelerator cards for sending external compressed files are those other than the first accelerator card.

[0093] Accordingly, in one embodiment, the external compressed file can also be decompressed to obtain external data blocks during the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card.

[0094] For example, such as Figure 4 The diagram shown is a timing diagram of the communication of the accelerator card provided in this embodiment. Taking the accelerator card starting from time 0 and sequentially performing asynchronous reception, compression, asynchronous transmission, decompression, and reduction operations as an example, firstly, between time 0 and T1, while performing asynchronous reception (receiving an external compressed file sent by any accelerator card), a single data block to be communicated is compressed. Between time 1 and T2, while asynchronously transmitting the compressed file, the external compressed file is decompressed to obtain the external data block. Finally, between time 2 and T3, the external data block is reduced, such as performing summation or averaging calculations according to preset rules. This achieves simultaneous execution of asynchronous reception, asynchronous transmission, and compression / decompression, thereby improving communication efficiency and shortening the overall training time of the model.

[0095] The model training time optimization method provided in this application involves: acquiring the global identifier and parallel dimension information of multiple training processes of the model to be trained; for any training process, determining the target group identifier of the training process based on the global identifier and parallel dimension information; dividing the multiple training processes into a corresponding number of local process groups based on the target group identifiers of each training process; allocating each local process group to a corresponding node to run the training processes in the local process group based on the node; acquiring the communication data to be communicated generated by the first accelerator card in the node in response to the training process; dividing the communication data to be communicated into multiple communication data blocks, compressing each communication data block at a time to obtain a communication compressed file; during the process of the first accelerator card sending the communication compressed file to the second accelerator card, compressing the uncompressed communication data blocks to be communicated, so that the compression process of the communication data blocks and the communication process of the communication compressed file overlap in time. The method described above determines the target group identifier of a training process based on its global identifier and parallel dimension information. Then, based on the target group identifiers of each training process, multiple training processes are divided into a corresponding number of local process groups. The process of determining the target group identifier for each training process is independent, allowing multiple training processes to determine local process groups in parallel. This shortens the time for dividing local process groups and consequently reduces training startup time. Furthermore, during accelerator card communication, the compression process of the data block to be communicated and the communication process of the compressed file to be communicated overlap to a certain extent, improving the communication efficiency between accelerator cards and thus reducing the overall model training time. Moreover, combining rapid process group construction with low-latency ensemble communication (layered hybrid communication backend) effectively reduces the time overhead of parallel training startup while ensuring the communication efficiency of parallel model training. Building upon this, a compression-based ensemble communication optimization method is introduced, which, combined with the communication characteristics of the accelerator card, overlaps compression and decompression with communication time, further effectively reducing communication overhead.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0097] Embodiments of this application also provide a model training time optimization apparatus for executing the model training time optimization method provided in the above embodiments.

[0098] like Figure 5 The diagram shown is a schematic representation of the model training time optimization device provided in this embodiment of the application. The model training time optimization device 50 includes: a first acquisition module 501, a determination module 502, a partitioning module 503, a training module 504, a second acquisition module 505, a compression module 506, and a communication module 507.

[0099] The system comprises the following modules: a first acquisition module for acquiring global identifiers and parallel dimension information of multiple training processes of the model to be trained; a determination module for determining the target group identifier of any training process based on its global identifier and parallel dimension information; a partitioning module for dividing multiple training processes into a corresponding number of local process groups based on the target group identifiers of each training process; a training module for assigning each local process group to a corresponding node to run the training processes within the local process group on the node; a second acquisition module for acquiring the communication data generated by the first accelerator card in response to the training process; a compression module for dividing the communication data into multiple communication data blocks and compressing each communication data block individually to obtain a communication compressed file; and a communication module for compressing uncompressed communication data blocks during the process of the first accelerator card sending the communication compressed file to the second accelerator card, so that the compression process of the communication data blocks and the communication process of the communication compressed file overlap in time.

[0100] For a description of the features in the embodiment corresponding to the model training time optimization device, please refer to the relevant description of the embodiment corresponding to the model training time optimization method, which will not be repeated here.

[0101] Embodiments of this application also provide an electronic device, such as... Figure 6 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described embodiments of the model training time optimization method.

[0102] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model training time optimization method when running.

[0103] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0104] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the model training time optimization method.

[0105] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the model training time optimization method.

[0106] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] The above provides a detailed description of a model training time optimization method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for optimizing model training time, characterized in that, include: Obtain the global identifiers and parallel dimension information of multiple training processes of the model to be trained; For any of the training processes, the target group identifier of the training process is determined based on the global identifier and parallel dimension information of the training process. Based on the target group identifier of each training process, the plurality of training processes are divided into a corresponding number of local process groups; Each of the local process groups is assigned to a corresponding node so that the training process in the local process group can be run on the node. Obtain the communication data generated by the first accelerator card in the node in response to the training process; The data to be communicated is divided into multiple data blocks to be communicated, and each individual data block to be communicated is compressed to obtain a compressed file to be communicated. During the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card, the uncompressed data block to be communicated is compressed so that the compression process of the data block to be communicated and the communication process of the compressed file to be communicated overlap in time. For any of the training processes, determining the target group identifier of the training process based on its global identifier and parallel dimension information includes: The position of the training process in the pipeline parallel group is determined based on the global identifier, data parallelism, and tensor parallelism of the training process. The data parallel group identifier of the training process is determined based on the position of the training process in the pipeline parallel group, the global identifier, the data parallelism, and the tensor parallelism. Wherein, the parallel dimension information includes the data parallelism and tensor parallelism, and the target group identifier includes the data parallelism group identifier; Determining the position of the training process within the pipelined parallel group based on the global identifier, data parallelism, and tensor parallelism of the training process includes: The position of the training process within the pipeline parallel group is determined based on the following formula: in, This indicates the position of the training process within the pipeline parallel group. Indicates the global identifier, This indicates the amount of data parallelism. This represents the parallel quantity of the tensor; The step of determining the pipeline parallel group identifier of the training process based on its position in the pipeline parallel group, the global identifier, the data parallelism, and the tensor parallelism includes: The data parallel group identifier of the training process is determined based on the following formula: in, The data parallel group identifier represents the training process.

2. The model training time optimization method according to claim 1, characterized in that, After dividing the plurality of training processes into a corresponding number of local process groups based on the target group identifier of each training process, the method includes: For any of the local process groups, the training processes are sorted in the local process group according to the global identifier of each training process.

3. The model training time optimization method according to claim 1, characterized in that, Local process groups assigned to different nodes communicate with each other via a hierarchical hybrid communication backend; The layered hybrid communication backend includes a strategy optimization layer, a transmission optimization layer, and a basic communication layer. The strategy optimization layer is used to select the optimal communication strategy, the transmission optimization layer is used to select the optimal communication channel, and the basic communication layer is used to perform data communication according to the optimal communication strategy and the optimal communication channel.

4. The model training time optimization method according to claim 1, characterized in that, The step of dividing the data to be communicated into multiple data blocks includes: The target size of the data block to be communicated is determined based on the preset accelerator card cache space size; According to the target size of the data block to be communicated, the data to be communicated is divided into multiple data blocks to be communicated.

5. The model training time optimization method according to claim 1, characterized in that, The method further includes: During the process of the first accelerator card receiving an external compressed file sent by any accelerator card, a single block of data to be communicated is compressed.

6. The model training time optimization method according to claim 5, characterized in that, The method further includes: During the process of the first accelerator card sending the compressed file to be communicated to the second accelerator card, the external compressed file is decompressed to obtain external data blocks.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the model training time optimization method as described in any one of claims 1 to 6 when executing the computer program.

Images