Cross-data-center distributed training method and device and computer program product

Abstract

The invention provides a cross-data-center distributed training method and device, electronic equipment, a storage medium and a computer program product, relates to the technical field of artificial intelligence, in particular to the technical fields of distributed training, large models and the like, and can be applied to distributed training scenes. According to the specific implementation scheme, for each parallel computing strategy in a plurality of parallel computing strategies, computing units in a plurality of data centers are grouped according to a grouping scale corresponding to the parallel computing strategy, and unit groups corresponding to the parallel computing strategies are obtained; determining a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by the plurality of parallel computing strategies to the communication delay duration; in the distributed training process of the to-be-trained model, controlling a unit group corresponding to the target parallel computing strategy, interacting communication data generated by the target parallel computing strategy among the plurality of data centers, and controlling unit groups corresponding to other parallel computing strategies to interact communication data generated by the target parallel computing strategy among the plurality of data centers; and interacting communication data generated by other parallel computing strategies in the affiliated data center. According to the invention, on the basis of maintaining the communication efficiency, the training efficiency of the distributed training process is improved based on a plurality of data centers and a plurality of parallel computing strategies.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as distributed training and large models, and in particular to a distributed training method, device, electronic device, storage medium and computer program product across data centers, which can be applied in distributed training scenarios. Background Art

[0002] For deep learning models, especially for large-scale models such as GPT-3, Megatron-LM, LLama 3.1, etc., distributed training has become a common training method. Distributed training allows the use of multiple computing nodes (usually in a distributed cluster) to jointly train a model, thereby accelerating the training process. With the continuous development of large models, the size of the GPU (Graphics Processing Unit) cluster required for model training continues to increase. In normal mode, the GPU is in a single data center (computer room), which can effectively reduce communication overhead. However, in reality, due to site limitations and electricity costs, the GPU cluster of a data center cannot be expanded indefinitely. During the training of large models, it is necessary to cross multiple data centers in physical space. Long-distance communication between data centers will inevitably increase network latency. Summary of the invention

[0003] The present disclosure provides a distributed training method, apparatus, electronic device, storage medium and computer program product across data centers.

[0004] According to a first aspect, a distributed training method across data centers is provided, including: for each of a plurality of parallel computing strategies, computing units in a plurality of data centers are grouped according to a grouping scale corresponding to the parallel computing strategy to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to divide a model to be trained and / or a training sample set to perform distributed training on the model to be trained; a target parallel computing strategy is determined from the plurality of parallel computing strategies according to the sensitivity of communication data generated by each of the plurality of parallel computing strategies to a communication delay duration; in a distributed training process of the model to be trained, the unit group corresponding to the target parallel computing strategy is controlled to exchange the communication data generated by the target parallel computing strategy between the plurality of data centers, and the unit groups corresponding to other parallel computing strategies are controlled to exchange the communication data generated by the other parallel computing strategies within the data center to which they belong.

[0005] According to the second aspect, a distributed training device across data centers is provided, including: a grouping unit, configured to group the computing units in multiple data centers for each of multiple parallel computing strategies according to the grouping scale corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to divide the model to be trained and / or the training sample set for distributed training of the model to be trained; a strategy determination unit, configured to determine a target parallel computing strategy from multiple parallel computing strategies according to the sensitivity of the communication data generated by each of the multiple parallel computing strategies to the communication delay duration; a communication control unit, configured to control the unit group corresponding to the target parallel computing strategy during the distributed training process of the model to be trained, to exchange the communication data generated by the target parallel computing strategy between multiple data centers, and to control the unit groups corresponding to other parallel computing strategies, to exchange the communication data generated by other parallel computing strategies within the data center to which they belong.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] According to the technology disclosed in the present invention, a cross-data center distributed training method and device are provided, which divides the computing units in multiple data centers based on the grouping scales corresponding to multiple parallel computing strategies, improves the rationality of the unit group, and helps to improve the data communication efficiency based on the unit group; based on the sensitivity of the communication data of multiple parallel computing strategies to the communication delay duration, controls the cross-data center interaction of sensitive communication data, controls the interaction of non-sensitive communication data within the data center, and reduces the impact of communication delays between data centers on the training process. Thus, on the basis of maintaining communication efficiency, based on multiple data centers and multiple parallel computing strategies, the training efficiency of the distributed training process is improved.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a distributed training method across data centers according to the present disclosure;

[0014] Figure 3 Schematic diagram of computing performance of the data parallel strategy and pipeline parallel strategy under different grouping scales according to this embodiment;

[0015] Figure 4 is a schematic diagram of grouping of data centers according to this embodiment;

[0016] Figure 5 is a schematic diagram of an application scenario of the distributed training method across data centers according to this embodiment;

[0017] Figures 6A-6C is a schematic diagram of communication data changes according to this embodiment;

[0018] Figure 7 is a flow chart of an embodiment of a data recommendation method according to the present disclosure;

[0019] Figure 8 is a structural diagram of an embodiment of a distributed training device across data centers according to the present disclosure;

[0020] Fig. 9 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 An exemplary architecture 100 is shown to which the cross-data center distributed training method and apparatus of the present disclosure can be applied.

[0024] like Figure 1 As shown, the system architecture 100 may include data centers 101, 102, 103, a network 104, and a server 105. The communication connection between the data centers 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for communication links between the data centers 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0025] The data centers 101, 102, and 103 include multiple computing units, which may be hardware devices or software that support network connection for data interaction and data processing. When the computing unit is hardware, it may be various electronic devices that support network connection, information acquisition, interaction, processing and other functions, including but not limited to GPU, CPU (Central Processing Unit), etc. When the computing unit is software, it may be installed in the electronic devices listed above. It may be implemented as multiple software or software modules for providing distributed services, or it may be implemented as a single software or software module. No specific limitation is made here.

[0026] The server 105 may be a server that provides various services, for example, a background processing server that uses multiple parallel computing strategies to perform distributed training on the training model using the computing units in the data centers 101, 102, and 103. As an example, the server 105 may be a cloud server.

[0027] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] It should also be noted that the cross-data center distributed training method provided by the embodiments of the present disclosure is generally executed by a server. Accordingly, all parts (eg, various units) included in the cross-data center distributed training device are generally all arranged in the server.

[0029] It should be understood that Figure 1 The number of data centers, networks, and servers in the system architecture is only illustrative. Depending on the implementation requirements, there may be any number of data centers, networks, and servers. When the electronic device on which the cross-data center distributed training method runs does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., server) on which the cross-data center distributed training method runs.

[0030] Please refer to Figure 2 , Figure 2 A flow chart of a distributed training method across data centers provided in an embodiment of the present disclosure. In process 200, the following steps are included:

[0031] Step 201 , for each of the multiple parallel computing strategies, group computing units in multiple data centers according to the grouping scale corresponding to the parallel computing strategy to obtain a unit group corresponding to the parallel computing strategy.

[0032] In this embodiment, the execution body of the distributed training method across data centers (for example, Figure 1 The server in the plurality of parallel computing strategies may group the computing units in the plurality of data centers according to the grouping scale corresponding to the parallel computing strategy for each of the plurality of parallel computing strategies, and obtain a unit group corresponding to the parallel computing strategy. The parallel computing strategy is used to divide the model to be trained and / or the training sample set for distributed training of the model to be trained. The model to be trained is generally a large language model, such as GPT (Generative Pre-trained Transformer), Megatron-LM (Language Model), LLama, and other models with large parameter scales.

[0033] The multiple parallel computing strategies can be at least two parallel computing strategies in a parallel computing strategy set, and the parallel computing strategy set includes but is not limited to DP (Data Parallelism) strategy, PP (Pipeline Parallelism) strategy, TP (Tensor Parallelism) strategy, EP (Expert Parallelism) strategy, and variations or mixtures of the above parallel computing strategies.

[0034] The data parallel strategy refers to dividing the dataset into multiple subsets, each of which is assigned to a computing unit cluster (or computing unit). The computing unit cluster trains its own deployed model copy with the received subset. Data parallelism focuses on the partitioning of the dataset and the synchronization of model copies. In data parallelism, each computing unit cluster maintains a complete copy of the model and independently calculates the gradient of its specified batch. Gradient synchronization is a key step in data parallelism, which can be achieved through a parameter server or a decentralized AllReduce algorithm.

[0035] The pipeline parallel strategy is a model parallel strategy that implements a pipelined computing process by dividing the different layers of the model and assigning different layers to different computing unit clusters (or computing units). Pipeline parallelism focuses on the computing flow between model layers and divides the different Transformer layers of the model. In pipeline parallelism, each computing unit is responsible for a part of the model layer, and data flows between devices by layer for forward and backward propagation. Pipeline parallelism can improve device utilization and reduce the idle time of devices through the MicroBatch pipeline parallel solution.

[0036] The tensor parallel strategy is to split the tensors in the model and assign different parts of the tensors to different computing unit clusters (or computing units) to achieve parallel computing. Tensor parallelism mainly focuses on tensor operations within the model, which is to split the specific matrix operations within the layer. It needs to solve the problem of how to split the parameters to different devices and how to ensure mathematical consistency after splitting. In tensor parallelism, each computing unit cluster (or computing unit) usually only saves part of the model's parameters and exchanges data with other devices through communication links.

[0037] In the expert parallel strategy, the model is divided into multiple "experts" or sub-models, each of which is responsible for processing a portion of the model's input or output, or for performing a specific task in the model. These experts are assigned to different computing devices to achieve parallel computing. Through expert parallelism, the memory requirements of a single computing device can be further reduced, the graphics memory efficiency can be improved, and the training process can be accelerated.

[0038] Multiple data centers refer to data centers with at least two data centers. Multiple data centers are generally data centers with a certain distance between them in physical space, but it is not ruled out that there is no distance between multiple data centers in physical space. Each data center includes multiple computing units. Generally, each data center includes a large number of computing units. Computing units are, for example, GPUs, CPUs, etc.

[0039] As an example, the above-mentioned execution entity can determine the grouping scale corresponding to each of the multiple parallel computing strategies based on the received grouping scale indication instruction; for each of the multiple parallel computing strategies, the computing units in the multiple data centers are grouped according to the grouping scale corresponding to the parallel computing strategy to obtain the unit group corresponding to the parallel computing strategy.

[0040] As another example, the above-mentioned execution entity can determine the model by grouping scale, and determine the grouping scale corresponding to each of the multiple parallel computing strategies according to the characteristics of the model to be trained (such as model structure, parameter scale, etc.) and the number of computing units in multiple data centers; for each of the multiple parallel computing strategies, according to the grouping scale corresponding to the parallel computing strategy, the computing units in the multiple data centers are grouped to obtain the unit group corresponding to the parallel computing strategy.

[0041] For example, multiple data centers include data centers DC1 and DC2, and the number of computing units in each data center is 256; multiple parallel computing strategies include strategies P1, P2 and P3, and strategies P1, P2 and P3 correspond to grouping sizes of 4, 8, and 32 respectively. Then, the 512 computing units in data centers DC1 and DC2 are divided into 128 (512÷4) unit groups corresponding to strategy P1, 64 (512÷8) unit groups corresponding to strategy P2, and 16 (512÷32) unit groups corresponding to strategy P3.

[0042] In some optional implementations of this embodiment, the multiple parallel computing strategies include data parallel strategy, pipeline parallel strategy and tensor parallel strategy. For example, in this implementation, the PTD (Parallel Task Decomposition) parallel Megatron segmentation framework is adopted.

[0043] In this implementation, the execution subject can determine the grouping scales corresponding to the multiple parallel computing strategies in the following manner:

[0044] For the tensor parallel strategy, the grouping scale corresponding to the tensor parallel strategy is determined according to the computing architecture of the service device where the computing unit is located.

[0045] Among the three parallel computing strategies mentioned above, the tensor parallel strategy generates the largest amount of communication data, requiring high-speed, low-latency communication interactions between multiple GPUs in the same unit group. Taking the Nvlink architecture currently used in the industry as an example, it is a high-speed, low-latency inter-GPU communication interface that allows GPUs to exchange data directly without going through the CPU or other intermediate devices, thereby enhancing communication between GPUs, improving performance, and achieving seamless parallel processing. Limited by the Nvlink architecture, the 8 GPUs in the server can communicate quickly. Therefore, the group size corresponding to the tensor parallel strategy can be determined as 8.

[0046] It should be noted that based on the development of science and technology, the computing architecture and communication architecture of service equipment will be updated iteratively. Based on this, the grouping scale corresponding to the tensor parallel strategy can be updated along with the iterative updates of the computing architecture and communication structure of the service equipment.

[0047] For the data parallel strategy and the pipeline parallel strategy, the grouping scale corresponding to the data parallel strategy is determined to be larger than the grouping scale corresponding to the pipeline parallel strategy, so as to determine the grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy respectively.

[0048] Continue to refer Figure 3 , which shows a schematic diagram of the computing performance of data parallel strategy and pipeline parallel strategy under different grouping sizes.

[0049] When the batch size is 512, as the group size corresponding to the pipeline parallel strategy increases, the group size corresponding to the data parallel strategy decreases, and the computing performance of each GPU decreases. TeraFLOP / s means trillion floating-point operations per second.

[0050] When the batch size is 32, as the group size corresponding to the pipeline parallel strategy increases, the group size corresponding to the data parallel strategy decreases, and the computing performance of each GPU also decreases.

[0051] Therefore, in this implementation, the grouping scale corresponding to the data parallel strategy can be determined to be larger than the grouping scale corresponding to the pipeline parallel strategy, so as to determine the grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy respectively.

[0052] As an example, two data centers each include 8192 GPUs, and the grouping sizes corresponding to the data parallel strategy, pipeline parallel strategy, and tensor parallel strategy are 128, 16, and 8, respectively. That is, for each data center, it is divided into 64 unit groups corresponding to the data parallel strategy, 512 unit groups corresponding to the pipeline parallel strategy, and 1024 unit groups corresponding to the tensor parallel strategy. Among them, the unit groups corresponding to the data parallel strategy include 16 unit groups corresponding to the tensor parallel strategy and 8 unit groups corresponding to the pipeline parallel strategy.

[0053] In this implementation, a specific method for determining the grouping size corresponding to each of multiple parallel computing strategies is provided, which helps to further improve the rationality of grouping and thereby improve the communication efficiency and rate during distributed training.

[0054] Step 202 : determining a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by the plurality of parallel computing strategies to the communication delay duration.

[0055] In this embodiment, the execution subject may determine a target parallel computing strategy from a plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration.

[0056] The communication data generated by the parallel computing strategy refers to the data that needs to be synchronized between computing units to run the parallel computing strategy. Specifically, for the data parallel strategy, the communication data generated mainly involves the model parameters or gradient data that the GPU needs to synchronize to ensure that different model copies of the model to be trained are consistent. For the pipeline parallel strategy, the communication data generated refers to the data transmitted between different stages of the pipeline. For example, when a GPU completes the part of the calculation it is responsible for, it needs to pass the result to the next GPU for subsequent calculations. For the tensor parallel strategy, the communication data generated is the data related to the model weights or calculation graphs transmitted between GPUs. Specifically, it may include model parameters or gradient information that need to be synchronized or exchanged between GPUs during the training process.

[0057] The higher the sensitivity of the communication data generated by the parallel computing strategy to the communication delay duration, the more the communication data generated by the parallel computing strategy can accept the communication delay, and the smaller the negative impact on the distributed training process (for example, reduced training efficiency); the higher the sensitivity of the communication data generated by the parallel computing strategy to the communication delay duration, the less the communication data generated by the parallel computing strategy can accept the communication delay, and the greater the negative impact on the distributed training process (for example, reduced training efficiency).

[0058] As an example, the execution subject or the electronic device connected to the execution subject in communication is provided with corresponding relationship data that characterizes the corresponding relationship between the parallel computing strategy and the sensitivity; after determining the parallel computing strategy adopted in the distributed training process of the model to be trained, its sensitivity can be determined through the corresponding relationship data; and the parallel computing strategy with the lowest sensitivity among multiple parallel computing strategies is determined as the target parallel computing strategy. In the historical distributed training process across data centers, the corresponding relationship data is determined by analyzing the communication efficiency of different parallel operation strategies.

[0059] As another example, the execution entity may perform sensitivity sorting on multiple parallel computing strategies through a sensitivity sorting model, thereby determining a parallel computing strategy with the lowest sensitivity among the multiple parallel computing strategies as the target parallel computing strategy.

[0060] In some optional implementations of this embodiment, the above-mentioned execution entity may perform the above-mentioned step 202 in the following manner: in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivity corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, the data parallel strategy is determined as the target parallel computing strategy.

[0061] The sensitivity of the data parallel strategy is lower than that of the tensor parallel strategy and pipeline parallel strategy respectively, mainly based on the following two points:

[0062] 1. The message size of the communication data generated by the pipeline parallel strategy is smaller, while the message size of the communication data generated by the data parallel strategy is larger. Communication data with large message size is less sensitive to long-distance communication across data centers.

[0063] The pipeline parallel strategy assigns different layers or stages of the model to different computing units for calculation, and data is passed between stages in a pipeline manner. Therefore, communication under the pipeline parallel strategy mainly occurs between adjacent computing units, and what is transmitted is the output of the previous layer (as the input of the next layer) and possible gradient information. Because the pipeline parallel strategy only needs to transfer data between adjacent devices instead of broadcasting it across the entire cluster, the amount of communication is relatively small. In addition, since the output of each layer or stage usually only contains the calculation results of that layer or stage, the size of this data is also relatively limited.

[0064] The data parallel strategy copies the entire model to multiple computing unit clusters (or computing units), each of which independently processes a portion of the data set and calculates the gradient. At the end of each round of training, gradient information needs to be exchanged between the computing unit clusters for synchronous updates. Because the data parallel strategy requires a large communication at the end of each round to synchronize the gradient information of all device clusters. This gradient information involves all parameters of the model, so the amount of communication is usually large. In addition, as the model size increases and the number of parameters increases, the size of the gradient information will also increase accordingly.

[0065] 2. In the data parallel processing process under the data parallel strategy, the communication (data transmission or synchronization) process and the computing process are effectively overlapped or executed in parallel, which can maximize resource utilization, reduce the overall computing time, and improve the efficiency of parallel computing.

[0066] In data parallelism, the model is replicated on multiple computing unit clusters, each of which processes a subset of the input data. Ideally, while a computing unit cluster is performing calculations, other computing unit clusters can simultaneously transmit data or prepare data for the next computing task. This overlap can significantly reduce the total computing time because communication and computation are no longer serial, but can be performed in parallel. Therefore, the overlap of the communication process and the computation process in the data parallel process can, to a certain extent, mask the problems caused by increased network latency.

[0067] In this implementation, for a mixed parallel scenario of data parallel strategy, pipeline parallel strategy and tensor parallel strategy, the data parallel strategy is determined as the target parallel strategy, which helps to further reduce the impact of communication delays between data centers on the training process.

[0068] Step 203, during the distributed training process of the training model, controls the unit group corresponding to the target parallel computing strategy to exchange the communication data generated by the target parallel computing strategy between multiple data centers, and controls the unit groups corresponding to other parallel computing strategies to exchange the communication data generated by other parallel computing strategies within the data center to which they belong.

[0069] In this embodiment, the above-mentioned execution entity can control the unit group corresponding to the target parallel computing strategy during the distributed training process of the training model, exchange the communication data generated by the target parallel computing strategy between multiple data centers, and control the unit groups corresponding to other parallel computing strategies, and exchange the communication data generated by other parallel computing strategies within the data center to which they belong.

[0070] Specifically, for multiple unit groups corresponding to the target parallel computing strategy, data synchronization is required between the multiple unit groups. Therefore, the communication data generated by the target parallel computing strategy needs to be exchanged between multiple data centers. For unit groups corresponding to other parallel computing strategies, data communication only needs to be performed within the unit group. Therefore, the communication data generated by the target parallel computing strategy only needs to be exchanged within the data center.

[0071] In some optional implementations of this embodiment, the execution subject may perform the process of "controlling the unit group corresponding to the target parallel computing strategy to exchange the communication data generated by the target parallel computing strategy between multiple data centers" in step 203 as follows:

[0072] The first step is to establish a plurality of first ring communication links between a plurality of unit groups corresponding to a target parallel computing strategy.

[0073] The first ring-shaped communication link includes a computing unit in each of the multiple unit groups, and some of the models to be trained included in the multiple computing units in the first ring-shaped communication link are the same.

[0074] In the second step, in the first ring communication link, a global reduction communication method is used to interact with the communication data generated by the target parallel computing strategy.

[0075] Continue to refer Figure 4 , showing a schematic diagram of data center grouping.

[0076] exist Figure 4 In the example, there are two data centers, DC1 and DC2, each of which includes 8192 GPUs. The group sizes corresponding to the data parallel strategy, pipeline parallel strategy, and tensor parallel strategy are 128, 16, and 8, respectively. The two data centers include a total of 128 unit groups corresponding to the data parallel strategy, namely DP 1-DP 128. Each DP group includes 16 unit groups corresponding to the tensor parallel strategy, namely TP 1-TP 16. Each TP group can be a computing node (such as a server), and each TP group includes 8 computing units, 1-8; each DP group includes 8 unit groups corresponding to the pipeline parallel strategy, namely PP 1-PP 8.

[0077] Each DP group includes complete parameters of a model to be trained, and each computing unit in a DP group includes parameters of a portion of the model to be trained.

[0078] The first ring communication link is, for example, Figure 4 The solid line in the figure represents the ring communication link and the dotted line in the figure represents the ring communication link. Figure 4The grouping method shown in the figure includes 128 first ring communication links in total. The thin solid line and the thin dotted line in the figure represent the communication between two data centers.

[0079] The global reduction communication method is the All Reduce communication method, specifically the Ring All Reduce communication method. Ring All Reduce is divided into three steps: Split, ScatterReduce, and AllGather.

[0080] 1. Split stage: According to the size N of the cluster, divide the data to be synchronized into N data blocks evenly.

[0081] 2. ScatterReduce phase: Perform N-1 ScatterReduce iterations. In each iteration, the GPU will send a data block to its right neighbor and receive a data block from its left neighbor and accumulate the received data blocks into the local data block corresponding to the received data block. Each node exchanges data in turn so that each node contains only a part (1 / N) of the final result. The blocks sent and received by each GPU are different in each iteration; the nth GPU starts by sending block N and receiving block N-1. In subsequent iterations, each iteration sends the data blocks accumulated based on the previous iteration. When ScatterReduce is completed, each node has a data block that accumulates the data of the corresponding data blocks of all other worker nodes.

[0082] 3. AllGather phase: Each node exchanges data again and finally gets the complete result. This process is the same as scatter-reduce (N-1 iterations of sending and receiving), except that the values ​​received by the GPU are not accumulated, but simply overwritten. The nth GPU first sends the n+1th block and receives the nth block, and then always sends the data block it just received in subsequent iterations.

[0083] After the AllGather operation is completed, all data blocks of all worker nodes contain data from other worker nodes.

[0084] In this implementation, a specific communication method for a unit group corresponding to a target parallel computing strategy is provided, which, combined with a ring communication link and a global reduction communication method, helps to further reduce the impact of communication delays between data centers on the training process.

[0085] In some optional implementations of this embodiment, the execution subject may perform the second step in the following manner:

[0086] First, according to the transmission rate between the computing units in the first ring communication link and the preset redundancy multiple, the target transmission rate of the communication channel between the plurality of data centers is determined.

[0087] The preset redundancy multiple can be specifically set according to actual conditions, for example, the preset redundancy multiple is 2. As an example, the target transmission rate is at least a preset redundancy multiple of the transmission rate between the computing units in the first ring communication link.

[0088] Assuming that the transmission rate of the remote network interface card (RNIC) of the computing unit is 400G, then in each first ring communication link, 800G bandwidth can be configured across data centers. This is because when the computing nodes meet the network affinity, there is only one pair of communication flows across data centers in each first ring communication link, so 400G×2 bandwidth is required to support high-throughput communication of a single first ring communication link. In this way, the interconnection bandwidth of 128 first ring communication links between DC1 and DC2 will reach 102.4T (800G×128).

[0089] Then, in the first ring communication link, the communication data generated by the interactive target parallel computing strategy is transmitted using the global reduction communication mode and the target transmission rate.

[0090] In this implementation, a higher transmission rate is set for the communication channels between multiple data centers, which can avoid the problem of channel congestion and improve the communication efficiency of communication data across data centers.

[0091] In some optional implementations of this embodiment, the execution subject may execute the process of "adopting the target transmission rate and communicating the communication data generated by the target parallel computing strategy" in the following manner:

[0092] First, based on a preset capacity threshold, a target capacity of a buffer corresponding to a communication channel between multiple data centers is set.

[0093] The preset capacity threshold can be specifically set based on actual conditions and is not limited here.

[0094] In the communication process based on NCCL (NVIDIA Collective Communications Library), NCCL_BUFFSIZE determines the size of the buffer on each communication channel, which indirectly determines the slice size SliceSize of data transmission. Since data transmission needs to pass through the buffer, if the buffer is too small, the reading and writing of the buffer are more likely to be blocked. Especially in the long-distance cross-data center communication process, the network delay increases. At this time, if the traditional small-capacity buffer is used, all slices of a single chunk will wait for confirmation from the receiver after sending. The ideal multi-slice pipeline transmission mode degenerates into a single chunk sending mode waiting for confirmation and then sending again, resulting in reduced network transmission performance.

[0095] Through experiments, it is found that communication based on a large-capacity buffer (not less than the preset capacity threshold) can effectively improve the performance of NCCL All Reduce over long distances. Taking cross-data center communication within 100 kilometers as an example, the use of a large-capacity buffer can achieve a performance improvement of more than 200% compared to the default capacity (4M) buffer.

[0096] Of course, simply increasing the buffer capacity will result in more GPU memory being occupied. How to balance the relationship between buffer capacity and GPU memory needs to be judged based on actual conditions. However, test results show that in training long-distance communications across data centers, NCCL's large-capacity buffer communication is an effective way to optimize network communications and reduce performance losses.

[0097] Then, the communication data generated by the interactive target parallel computing strategy is adopted with the buffer of target transmission rate and target capacity.

[0098] In this implementation, by increasing the capacity of the buffer and combining it with an increased target transmission rate, the problem of channel congestion can be further avoided and the communication efficiency of communication data across data centers can be improved.

[0099] In some optional implementations of this embodiment, the execution subject may perform the process of "controlling the unit group corresponding to the target parallel computing strategy to exchange the communication data generated by the target parallel computing strategy between multiple data centers" in step 203 as follows:

[0100] For tensor parallel strategy:

[0101] First, a second ring communication link is established based on multiple computing units in the unit group corresponding to the tensor parallel strategy; then, in the second ring communication link, a global reduction communication method is adopted to exchange communication data generated by the tensor parallel strategy.

[0102] Continue to refer Figure 4 , for 1-8 computing units in each TP group, a second ring communication link is established; and then a global reduction communication method is adopted to exchange data within the TP group.

[0103] For pipeline parallel strategy:

[0104] A point-to-point communication method is adopted between multiple computing units in the unit group corresponding to the pipeline parallel strategy to exchange communication data generated by the pipeline parallel strategy.

[0105] Peer-to-peer communication allows two specific GPUs to exchange data directly.

[0106] Continue to refer Figure 4 ,For the 16 computing units in each PP group, in the forward computing process, the current computing unit receives the data of the previous computing unit, processes it, and transmits the processed data to the next computing unit; ,for the reverse computing process, the above opposite transmission direction is adopted, thereby realizing data communication between the computing units in the same PP.

[0107] In this implementation, a specific communication method for a unit group corresponding to other parallel computing strategies is provided. Different parallel computing strategies adopt their appropriate communication methods, thereby improving the data communication efficiency within the unit group corresponding to other parallel computing strategies.

[0108] Continue to see Figure 5 , Figure 5 It is a schematic diagram 500 of an application scenario of the distributed training method across data centers according to the present embodiment. Server 501 needs to control the computing units in two data centers 502 and 503 to perform distributed training on the model to be trained. First, for each of the multiple parallel computing strategies, the computing units in the multiple data centers are grouped according to the grouping scale corresponding to the parallel computing strategy to obtain the unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to divide the model to be trained and / or the training sample set to distribute the training model to be trained; then, according to the sensitivity of the communication data generated by each of the multiple parallel computing strategies to the communication delay time, the target parallel computing strategy is determined from the multiple parallel computing strategies; finally, in the distributed training process of the model to be trained, the unit group corresponding to the target parallel computing strategy is controlled to exchange the communication data generated by the target parallel computing strategy between the multiple data centers, and the unit groups corresponding to other parallel computing strategies are controlled to exchange the communication data generated by other parallel computing strategies within the data center to which they belong.

[0109] In this embodiment, a cross-data center distributed training method is provided, which divides the computing units in multiple data centers based on the grouping scales corresponding to multiple parallel computing strategies, improves the rationality of the unit group, and helps to improve the data communication efficiency based on the unit group; based on the sensitivity of the communication data of multiple parallel computing strategies to the communication delay duration, controls the cross-data center interaction of sensitive communication data, controls the interaction of non-sensitive communication data within the data center, and reduces the impact of communication delays between data centers on the training process. Thus, on the basis of maintaining communication efficiency, based on multiple data centers and multiple parallel computing strategies, the training efficiency of the distributed training process is improved.

[0110] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operations: for the target computing unit that performs data exchange between multiple data centers in the unit group corresponding to the target parallel computing strategy, turn off the adaptive routing function of the target computing unit.

[0111] After enabling AR (adaptive routing), AR needs to handle out-of-order message reception. The processing mechanism here uses a bitmap method, so a relatively small sending window is required. If the sending window is too large, the processing complexity will increase. Therefore, the network card lowers the size of the sending window (which is a speed reduction behavior in itself). After the sending window is reduced, the frequency of waiting for ACK (acknowledgement) messages will increase. In addition, the RTT (Round-Trip Time) delay increases during long-distance communication across data centers. The overall superposition further reduces the message sending rate.

[0112] Experiments have shown that the number of messages that need to be confirmed increases dramatically after AR is enabled. For example, when capturing 10,000 messages, 676 messages need to be confirmed after AR is enabled, and 46 messages need to be confirmed after AR is disabled, a sharp decrease in the number of messages.

[0113] If AR is enabled in long-distance RDMA (Remote Direct Memory Access), the following situations will occur: (1) The message needs to carry an additional RETH (Remote Extended Transport Header) extension header, which causes the payload to become smaller and the efficiency to decrease. (2) The message receiver needs to process the received out-of-order messages, so it will reduce the sending window size to facilitate the processing of out-of-order message delivery, thereby reducing the sending rate of the sender. (3) The number of messages that need to be confirmed increases. (4) The long-distance RTT increases, and the number of messages that need ACK increases, which further prolongs the message confirmation waiting time and reduces the sending efficiency.

[0114] In this implementation, by disabling the adaptive routing function of the target computing unit, the above situation can be effectively avoided and the communication efficiency of the communication data can be improved.

[0115] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operation: for a preset collective communication library on which the communication process between multiple data centers is based, disable the adaptive routing function of the preset collective communication library.

[0116] An example of a preset collective communication library is NCCL, which uses RDMA technology, an efficient way to transfer data that allows computers in a network to directly access each other's memory without intervention from the operating system or CPU. Through RDMA, NCCL is able to achieve low-latency, high-bandwidth data transfer. NCCL supports both point-to-point communication and cluster communication modes. Point-to-point communication allows data to be exchanged directly between two specific GPUs, while cluster communication involves data exchange between multiple GPUs. NCCL selects the most appropriate communication mode based on specific communication requirements to achieve efficient data transfer. NCCL is performance optimized for NVIDIA GPUs and networks, and can achieve high bandwidth and low latency on high-speed interconnect networks such as PCIe (Peripheral Component Interconnect express) and NVLink. This hardware acceleration enables NCCL to perform well in scenarios that require a large amount of inter-GPU communication, such as deep learning training. When configuring the NCCL library, turning off AR can further improve the communication efficiency of communication data.

[0117] In some optional implementations of this embodiment, the above execution subject may also perform the following operations:

[0118] First, the performance degradation ratio of the long-distance communication process between multiple data centers generated by the target parallel computing strategy is determined compared with the short-distance communication process within a data center.

[0119] In this implementation, the above-mentioned execution entity can use the performance testing tool nccl_test in the NCCL library to measure the communication performance under different network conditions (such as different distances, different bandwidths), so as to evaluate the impact of network latency and bandwidth on training performance.

[0120] Taking the performance of a short-distance communication process within a data center as 100% as an example, the performance degradation ratio information is, for example, 316%.

[0121] Then, the proportion of the duration of the communication process that is not covered by the computing process in the communication data generated by the target parallel computing strategy in an iterative training cycle is determined.

[0122] Continue to refer Figures 6A-6C , Fig. 6A The diagram in Figure 1 shows the communication data changes during a complete iteration cycle. Figure 6B A communication diagram showing the communication data generated by the data parallel strategy, Figure 6C Communication diagram showing communication data generated by the pipeline parallel strategy.

[0123] like Fig. 6A As shown in the figure, in a complete iteration cycle, there is only communication data corresponding to the PP strategy in the forward calculation, while there is communication data for the DP strategy and the PP strategy in the reverse calculation. After the reverse calculation is completed, there is a short period of communication data of the DP strategy that is not covered by the calculation, which fully proves that under the DP strategy, most of the communication process and the calculation process overlap. The communication data of the DP strategy is much higher than that of the PP strategy, which also fully demonstrates that the message size of the communication data of the DP strategy is much larger than that of the PP strategy. The traffic moment at the highest point of the curve is the communication data traffic of the communication process that is not covered by the calculation process, which is also the DP communication part that affects the overall performance of the distributed training process in the fast data center.

[0124] like Figure 6B , 6C As shown, the circled DP communication part corresponds to no PP traffic. It is visually estimated in the above figure that the DP part that is not covered by the calculation accounts for about 5% (relatively conservative) in the entire iteration cycle.

[0125] Finally, the performance loss information of the distributed training process is determined based on the performance degradation ratio information and the proportion information.

[0126] Continuing with the example that the performance degradation ratio is 316% and the proportion is 5%, the performance loss information is 316%*5%+95%-1=10.08%.

[0127] In this implementation, a performance estimation method in a cross-data center communication scenario is provided, which can quantify the performance loss in the distributed training process and provide the evaluation accuracy of the distributed training process.

[0128] Continue to refer Figure 7 , shows a schematic process 700 of another embodiment of a distributed training method across data centers according to the present disclosure. In process 700, the following steps are included:

[0129] Step 701, determining the grouping scale corresponding to the tensor parallel strategy according to the computing architecture of the service equipment where the computing units in the multiple data centers are located.

[0130] Step 702: determine the grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy respectively by adopting a determination method that the grouping scale corresponding to the data parallel strategy is larger than the grouping scale corresponding to the pipeline parallel strategy.

[0131] Step 703, for each parallel computing strategy among the data parallel strategy, the pipeline parallel strategy and the tensor parallel strategy, the computing units in the multiple data centers are grouped according to the grouping scale corresponding to the parallel computing strategy to obtain the unit group corresponding to the parallel computing strategy.

[0132] Among them, the parallel computing strategy is used to split the model to be trained and / or the training sample set to perform distributed training on the model to be trained.

[0133] Step 704 , in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivity corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, the data parallel strategy is determined as the target parallel computing strategy.

[0134] Step 705: Establish a second ring communication link based on multiple computing units in the unit group corresponding to the tensor parallel strategy.

[0135] Step 706: In the second ring communication link, a global reduction communication method is used to exchange communication data generated by the tensor parallel strategy.

[0136] Step 707, a point-to-point communication method is used between multiple computing units in the unit group corresponding to the pipeline parallel strategy to exchange communication data generated by the pipeline parallel strategy.

[0137] Step 708: Establish multiple first ring communication links between multiple unit groups corresponding to the data parallel computing strategy.

[0138] The first ring-shaped communication link includes a computing unit in each of the multiple unit groups, and some of the models to be trained included in the multiple computing units in the first ring-shaped communication link are the same.

[0139] Step 709: In the first ring communication link, a global reduction communication method is used to exchange communication data generated by the data parallel computing strategy.

[0140] Process 700 of the cross-data center distributed training method in this embodiment specifically illustrates the process of determining the grouping scale of the parallel computing strategy, the process of determining the target parallel computing strategy, and the communication process of each parallel computing strategy. On the basis of maintaining communication efficiency, the training efficiency of the distributed training process is improved based on multiple data centers and multiple parallel computing strategies.

[0141] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a distributed training device across data centers. The system embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0142] like Figure 8 As shown, the distributed training device 800 across data centers includes: a grouping unit 801, which is configured to group the computing units in multiple data centers for each parallel computing strategy in multiple parallel computing strategies according to the grouping scale corresponding to the parallel computing strategy, so as to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to divide the model to be trained and / or the training sample set for distributed training of the model to be trained; a strategy determination unit 802, which is configured to determine the target parallel computing strategy from multiple parallel computing strategies according to the sensitivity of the communication data generated by each of the multiple parallel computing strategies to the communication delay duration; a communication control unit 803, which is configured to control the unit group corresponding to the target parallel computing strategy during the distributed training process of the model to be trained, to exchange the communication data generated by the target parallel computing strategy between multiple data centers, and to control the unit groups corresponding to other parallel computing strategies, to exchange the communication data generated by other parallel computing strategies within the data center to which they belong.

[0143] In some optional implementations of this embodiment, multiple parallel computing strategies include a data parallel strategy, a pipeline parallel strategy and a tensor parallel strategy, and the device also includes: a scale determination unit (not shown in the figure), configured to: determine the grouping scale corresponding to the tensor parallel strategy according to the computing architecture of the service device where the computing unit is located; adopt a determination method in which the grouping scale corresponding to the data parallel strategy is larger than the grouping scale corresponding to the pipeline parallel strategy, and determine the grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy respectively.

[0144] In some optional implementations of this embodiment, the strategy determination unit 802 is further configured to: in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivity corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, determine the data parallel strategy as the target parallel computing strategy.

[0145] In some optional implementations of the present embodiment, the communication control unit 803 is further configured to: establish multiple first ring communication links between multiple unit groups corresponding to the target parallel computing strategy, wherein the first ring communication link includes a computing unit in each of the multiple unit groups, and some of the models to be trained included in the multiple computing units in the first ring communication link are the same; in the first ring communication link, a global reduction communication method is adopted to exchange the communication data generated by the target parallel computing strategy.

[0146] In some optional implementations of this embodiment, the communication control unit 803 is further configured to: determine the target transmission rate of the communication channel between multiple data centers based on the transmission rate between the computing units in the first ring communication link and the preset redundancy multiple; in the first ring communication link, adopt a global reduction communication method and target transmission rate to interact with the communication data generated by the target parallel computing strategy.

[0147] In some optional implementations of this embodiment, the communication control unit 803 is further configured to: set a target capacity of a buffer corresponding to a communication channel between multiple data centers based on a preset capacity threshold; and use a buffer with a target transmission rate and a target capacity to interact with the communication data generated by the target parallel computing strategy.

[0148] In some optional implementations of this embodiment, the communication control unit 803 is further configured to: establish a second ring communication link based on multiple computing units in the unit group corresponding to the tensor parallel strategy; in the second ring communication link, adopt a global reduction communication method to exchange the communication data generated by the tensor parallel strategy; and adopt a point-to-point communication method between multiple computing units in the unit group corresponding to the pipeline parallel strategy to exchange the communication data generated by the pipeline parallel strategy.

[0149] In some optional implementations of this embodiment, the above-mentioned device also includes: a first setting unit (not shown in the figure), which is configured to turn off the adaptive routing function of the target computing unit for the target computing unit that performs data exchange between multiple data centers in the unit group corresponding to the target parallel computing strategy.

[0150] In some optional implementations of this embodiment, the above-mentioned device also includes: a second setting unit (not shown in the figure), which is configured to turn off the adaptive routing function of the preset collective communication library on which the communication process between multiple data centers is based.

[0151] In some optional implementations of the present embodiment, the calculation process and the communication process of the communication data corresponding to the target parallel computing strategy of the above-mentioned device are performed in an overlapping manner, and the above-mentioned device also includes: a loss determination unit (not shown in the figure), which is configured to: determine the performance degradation ratio information of the long-distance communication process between multiple data centers of the communication data generated by the target parallel computing strategy compared to the short-distance communication process within a data center; determine the proportion information of the duration of the part of the communication process not covered by the calculation process in the communication data generated by the target parallel computing strategy in an iterative training cycle; determine the performance loss information of the distributed training process based on the performance degradation ratio information and the proportion information.

[0152] In this embodiment, a distributed training device across data centers is provided, which divides the computing units in multiple data centers based on the grouping scales corresponding to multiple parallel computing strategies, improves the rationality of the unit group, and helps to improve the data communication efficiency based on the unit group; based on the sensitivity of the communication data of multiple parallel computing strategies to the communication delay duration, controls the cross-data center interaction of sensitive communication data, controls the interaction of non-sensitive communication data within the data center, and reduces the impact of communication delays between data centers on the training process. Thus, on the basis of maintaining communication efficiency, based on multiple data centers and multiple parallel computing strategies, the training efficiency of the distributed training process is improved.

[0153] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the distributed training method across data centers described in any of the above embodiments can be implemented when the at least one processor executes.

[0154] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the distributed training method across data centers described in any of the above embodiments when executed.

[0155] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the distributed training method across data centers described in any of the above embodiments.

[0156] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0157] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0158] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0159] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a distributed training method across data centers. For example, in some embodiments, the distributed training method across data centers may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the distributed training method across data centers described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform a distributed training method across data centers in any other appropriate manner (e.g., by means of firmware).

[0160] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable cross-data center distributed training device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0166] According to the technical solution of the embodiment of the present disclosure, a cross-data center distributed training method and device are provided, which divides the computing units in multiple data centers based on the grouping scales corresponding to multiple parallel computing strategies, improves the rationality of the unit group, and helps to improve the data communication efficiency based on the unit group; based on the sensitivity of the communication data of multiple parallel computing strategies to the communication delay duration, controls the cross-data center interaction of sensitive communication data, controls the interaction of non-sensitive communication data within the data center, and reduces the impact of communication delays between data centers on the training process. Thus, on the basis of maintaining communication efficiency, based on multiple data centers and multiple parallel computing strategies, the training efficiency of the distributed training process is improved.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A distributed training method across data centers, comprising: For each of the multiple parallel computing strategies, grouping computing units in multiple data centers according to the grouping scale corresponding to the parallel computing strategy to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to split the model to be trained and / or the training sample set to perform distributed training on the model to be trained; Determining a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration; During the distributed training process of the model to be trained, the unit group corresponding to the target parallel computing strategy is controlled to exchange the communication data generated by the target parallel computing strategy between the multiple data centers, and the unit groups corresponding to other parallel computing strategies are controlled to exchange the communication data generated by other parallel computing strategies within the data centers to which they belong.

2. The method according to claim 1, wherein: The plurality of parallel computing strategies include a data parallel strategy, a pipeline parallel strategy and a tensor parallel strategy, and The method further comprises: Determining a grouping scale corresponding to the tensor parallel strategy according to a computing architecture of a service device where the computing unit is located; The grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy are determined by adopting a determination method that the grouping scale corresponding to the data parallel strategy is larger than the grouping scale corresponding to the pipeline parallel strategy.

3. The method according to claim 2, wherein: The step of determining a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration comprises: In response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivities corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, the data parallel strategy is determined as the target parallel computing strategy.

4. The method according to any one of claims 1 to 3, wherein: The controlling the unit group corresponding to the target parallel computing strategy to exchange communication data generated by the target parallel computing strategy between the plurality of data centers includes: Establishing a plurality of first ring-shaped communication links between a plurality of unit groups corresponding to the target parallel computing strategy, wherein the first ring-shaped communication links include a computing unit in each of the plurality of unit groups, and a portion of the to-be-trained models included in the plurality of computing units in the first ring-shaped communication links are the same; In the first ring communication link, a global reduction communication method is adopted to exchange communication data generated by the target parallel computing strategy.

5. The method according to claim 4, wherein: The method of using a global reduction communication method in the first ring communication link to exchange communication data generated by the target parallel computing strategy includes: Determine a target transmission rate of the communication channels between the plurality of data centers according to the transmission rate between the computing units in the first ring communication link and a preset redundancy multiple; In the first ring communication link, a global reduction communication mode and the target transmission rate are adopted to exchange communication data generated by the target parallel computing strategy.

6. The method according to claim 5, wherein: Using the target transmission rate, exchanging communication data generated by the target parallel computing strategy, including: Based on a preset capacity threshold, setting a target capacity of a buffer corresponding to a communication channel between the plurality of data centers; The target transmission rate and the target capacity of the buffer are used to exchange communication data generated by the target parallel computing strategy.

7. The method according to claim 3, wherein: The control unit groups corresponding to other parallel computing strategies, and the communication data generated by other parallel computing strategies are exchanged within the corresponding data center, including: Establishing a second ring communication link based on multiple computing units in the unit group corresponding to the tensor parallel strategy; In the second ring communication link, a global reduction communication method is adopted to communicate the communication data generated by the tensor parallel strategy; and A point-to-point communication method is adopted between multiple computing units in a unit group corresponding to the pipeline parallel strategy to exchange communication data generated by the pipeline parallel strategy.

8. The method according to any one of claims 1 to 7, wherein: Also includes: For a target computing unit that performs data exchange between a plurality of the data centers in the unit group corresponding to the target parallel computing strategy, the adaptive routing function of the target computing unit is disabled.

9. The method according to claim 8, wherein: Also includes: For a preset collective communication library on which the communication process between the plurality of data centers is based, the adaptive routing function of the preset collective communication library is disabled.

10. The method according to any one of claims 1 to 9, wherein: The computing process and the communication process of the communication data corresponding to the target parallel computing strategy are performed in an overlapping manner, and The method further comprises: Determine information on a performance degradation ratio of a long-distance communication process between a plurality of data centers compared to a short-distance communication process within a data center in which communication data generated by the target parallel computing strategy is communicated; Determine the proportion of the duration of the part of the communication process not covered by the calculation process in the communication data generated by the target parallel computing strategy in an iterative training cycle; The performance loss information of the distributed training process is determined according to the performance degradation ratio information and the proportion information.

11. A distributed training device across data centers, comprising: A grouping unit is configured to group computing units in a plurality of data centers for each of a plurality of parallel computing strategies according to a grouping scale corresponding to the parallel computing strategy to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to divide a model to be trained and / or a training sample set to perform distributed training on the model to be trained; A strategy determination unit, configured to determine a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration; The communication control unit is configured to control the unit group corresponding to the target parallel computing strategy during the distributed training process of the model to be trained, to exchange the communication data generated by the target parallel computing strategy between the multiple data centers, and to control the unit groups corresponding to other parallel computing strategies, to exchange the communication data generated by other parallel computing strategies within the data centers to which they belong.

12. The device according to claim 11, wherein The plurality of parallel computing strategies include a data parallel strategy, a pipeline parallel strategy and a tensor parallel strategy, and The device also includes: The sizing unit is configured to: According to the computing architecture of the service device where the computing unit is located, the grouping scale corresponding to the tensor parallel strategy is determined; the grouping scale corresponding to the data parallel strategy is determined to be larger than the grouping scale corresponding to the pipeline parallel strategy, and the grouping scale corresponding to the data parallel strategy and the pipeline parallel strategy are determined respectively.

13. The device according to claim 12, wherein: The strategy determination unit is further configured to: In response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivities corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, the data parallel strategy is determined as the target parallel computing strategy.

14. The device according to any one of claims 11 to 13, wherein: The communication control unit is further configured to: Establish multiple first ring communication links between multiple unit groups corresponding to the target parallel computing strategy, wherein the first ring communication link includes a computing unit in each of the multiple unit groups, and some of the models to be trained included in the multiple computing units in the first ring communication link are the same; in the first ring communication link, a global reduction communication method is adopted to exchange communication data generated by the target parallel computing strategy.

15. The device according to claim 14, wherein: The communication control unit is further configured to: According to the transmission rate between the computing units in the first ring communication link and the preset redundancy multiple, the target transmission rate of the communication channel between the multiple data centers is determined; in the first ring communication link, the global reduction communication method and the target transmission rate are adopted to exchange the communication data generated by the target parallel computing strategy.

16. The device according to claim 15, wherein: The communication control unit is further configured to: Based on a preset capacity threshold, a target capacity of a buffer corresponding to a communication channel between the plurality of data centers is set; and communication data generated by the target parallel computing strategy is exchanged using the target transmission rate and the buffer of the target capacity.

17. The device according to claim 13, wherein: The communication control unit is further configured to: A second ring communication link is established based on a plurality of computing units in a unit group corresponding to the tensor parallel strategy; in the second ring communication link, a global reduction communication method is adopted to exchange communication data generated by the tensor parallel strategy; And a point-to-point communication method is adopted between multiple computing units in the unit group corresponding to the pipeline parallel strategy to exchange communication data generated by the pipeline parallel strategy.

18. The device according to any one of claims 11 to 17, wherein: Also includes: The first setting unit is configured to disable the adaptive routing function of a target computing unit that performs data exchange between the plurality of data centers in the unit group corresponding to the target parallel computing strategy.

19. The device according to claim 18, wherein: Also includes: The second setting unit is configured to disable the adaptive routing function of the preset collective communication library on which the communication process between the plurality of data centers is based.

20. The device according to any one of claims 11 to 19, wherein: The computing process and the communication process of the communication data corresponding to the target parallel computing strategy are performed in an overlapping manner, and The device also includes: The loss determination unit is configured to: Determine the performance degradation ratio information of the long-distance communication process between the multiple data centers generated by the target parallel computing strategy compared to the short-distance communication process within a data center; determine the proportion information of the duration of the part of the communication process not covered by the computing process in the communication data generated by the target parallel computing strategy in an iterative training cycle; determine the performance loss information of the distributed training process based on the performance degradation ratio information and the proportion information.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images