Distributed training method, device and computer program product across data centers

Abstract

The present disclosure provides a cross-data center distributed training method and device, electronic equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, specifically relates to the technical field of distributed training, large model and the like, and can be applied to the distributed training scene. The specific implementation scheme is: for each parallel computing strategy in a plurality of parallel computing strategies, grouping the computing units in the plurality of data centers according to the grouping size corresponding to the parallel computing strategy, to obtain the unit group corresponding to the parallel computing strategy; according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay time length, determining a target parallel computing strategy from the plurality of parallel computing strategies; in the distributed training process of the to-be-trained model, controlling the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between the plurality of data centers, and controlling the unit group corresponding to other parallel computing strategies to interact the communication data generated by other parallel computing strategies within the corresponding data center. The present disclosure improves the training efficiency of the distributed training process on the basis of maintaining the communication efficiency, based on the plurality of data centers and the plurality of parallel computing strategies.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of distributed training, large model, etc., and more particularly to a cross-data-center distributed training method and device, electronic equipment, storage medium and computer program product, which can be applied to a distributed training scenario. BACKGROUND

[0002] For deep learning models, especially in large-scale models such as GPT-3, Megatron-LM, LLama 3.1, etc., distributed training has become a common training method. Distributed training allows multiple computing nodes (usually within a distributed cluster) to jointly train a model, thereby accelerating the training process. With the continuous development of large models, the GPU (Graphics Processing Unit) cluster size required for model training is increasing. In the conventional mode, GPUs are within a single data center (machine room), which can effectively reduce communication overhead. However, in reality, due to site restrictions and power costs, the GPU cluster of a data center cannot be expanded indefinitely, and during large model training, it is necessary to span multiple data centers in physical space. Long-distance communication between data centers will inevitably increase network latency. SUMMARY

[0003] The present disclosure provides a cross-data-center distributed training method, device, electronic equipment, storage medium and computer program product.

[0004] According to a first aspect, a cross-data-center distributed training method is provided, comprising: for each parallel computing strategy in a plurality of parallel computing strategies, grouping computing units in a plurality of data centers according to a grouping size corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to split a to-be-trained model and / or a training sample set to train the to-be-trained model in a distributed manner; determining a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of communication data generated by each parallel computing strategy to communication delay time; and during the distributed training of the to-be-trained model, controlling the unit group corresponding to the target parallel computing strategy to interact communication data generated by the target parallel computing strategy between the plurality of data centers, and controlling the unit group corresponding to other parallel computing strategies to interact communication data generated by other parallel computing strategies within the corresponding data center.

[0005] According to a second aspect, a distributed training apparatus across data centers is provided, comprising: a grouping unit configured to, for each of a plurality of parallel computing strategies, group computing units in a plurality of data centers according to a grouping scale corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to split a to-be-trained model and / or a training sample set to train the to-be-trained model in a distributed manner; a strategy determining unit configured to determine a target parallel computing strategy from the plurality of parallel computing strategies according to a sensitivity of communication data generated by each of the plurality of parallel computing strategies to a communication delay duration; and a communication control unit configured to, in a process of training the to-be-trained model in a distributed manner, control the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between the plurality of data centers, and control unit groups corresponding to other parallel computing strategies to interact the communication data generated by the other parallel computing strategies within the data centers to which the unit groups correspond.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to perform the method described in any implementation manner of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0009] According to the technology of the present disclosure, a distributed training method and apparatus across data centers are provided, computing units in a plurality of data centers are divided based on grouping scales corresponding to a plurality of parallel computing strategies, the rationality of unit groups is improved, and the efficiency of data communication based on the unit groups is improved; based on the sensitivity of communication data of the plurality of parallel computing strategies to a communication delay duration, sensitive communication data is controlled to be interacted across data centers, and non-sensitive communication data is controlled to be interacted within data centers, and the influence of communication delay between data centers on a training process is reduced. Thus, on the basis of maintaining communication efficiency, the training efficiency of a distributed training process is improved based on a plurality of data centers and a plurality of parallel computing strategies.

[0010] It should be appreciated that the description set forth in this section is not intended to identify key or essential features of an embodiment of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be apparent from review of the disclosure, both hereinafter and in the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. Illustrations in the drawings are for purposes of illustrating an embodiment of the present application and are not intended to limit the present application.

[0012] Figure 1 is an exemplary system architecture diagram to which an embodiment according to the present disclosure can be applied;

[0013] Figure 2 is a flowchart of an embodiment of a distributed training method across data centers according to the present disclosure;

[0014] Figure 3 is a diagram of the computing performance of the data parallel strategy and the pipeline parallel strategy according to the present embodiment under different grouping scales;

[0015] Figure 4 is a grouping diagram of a data center according to the present embodiment;

[0016] Figure 5 is a diagram of an application scenario of a distributed training method across data centers according to the present embodiment;

[0017] Figures 6A-6C is a diagram of communication data changes according to the present embodiment;

[0018] Figure 7 is a flowchart of an embodiment of a data recommendation method according to the present disclosure;

[0019] Figure 8 is a structural diagram of an embodiment of a distributed training device across data centers according to the present disclosure;

[0020] Figure 9 is a structural diagram of a computer system suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. The descriptions of the exemplary embodiments of the present disclosure are intended to be illustrative, and not to be the only

[0022] The collection, storage, use, processing, transmission, provision and disclosure of the user personal information in the technical solutions of the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0023] Figure 1 An exemplary architecture 100 to which the cross-data-center distributed training method and device of the present disclosure can be applied is shown.

[0024] As shown in Figure 1 The system architecture 100 can include data centers 101, 102, 103, a network 104 and a server 105. The data centers 101, 102, 103 are communicatively connected to form a topology network, and the network 104 is a medium for providing a communication link between the data centers 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0025] The data centers 101, 102, 103 include a plurality of computing units, which can be hardware devices or software that support network connection for data interaction and data processing. When the computing unit is hardware, it can be various electronic devices that support network connection, information acquisition, interaction, processing, etc., including but not limited to GPU, CPU (Central Processing Unit), etc. When the computing unit is software, it can be installed in the above-mentioned electronic devices. It can be implemented as a plurality of software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made herein.

[0026] The server 105 can be a server that provides various services, for example, a background processing server that utilizes the computing units in the data centers 101, 102, 103 to perform distributed training on a to-be-trained model using a plurality of parallel computing strategies. As an example, the server 105 can be a cloud server.

[0027] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of a plurality of servers, or as a single server. When the server is software, it can be implemented as a plurality of software or software modules (for example, software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0028] It should also be noted that the cross-data-center distributed training method provided by the embodiments of the present disclosure is generally executed by a server. Accordingly, the cross-data-center distributed training device includes various parts (for example, various units) which are generally all arranged in the server.

[0029] It should be understood that Figure 1 The number of data centers, networks, and servers shown is merely illustrative. Any number of data centers, networks, and servers can be included depending on implementation needs. When the electronic devices running the distributed training method across data centers do not require data transfer with other electronic devices, the system architecture may consist only of the electronic devices (e.g., servers) running the distributed training method across data centers.

[0030] Please refer to Figure 2 , Figure 2 A flowchart illustrating a distributed training method across data centers provided in this disclosure embodiment. Flowchart 200 includes the following steps:

[0031] Step 201: For each of the multiple parallel computing strategies, the computing units in the multiple data centers are grouped according to the grouping size corresponding to the parallel computing strategy to obtain the unit group corresponding to the parallel computing strategy.

[0032] In this embodiment, the execution entity of the distributed training method across data centers (e.g., Figure 1 The server in the system can group computing units in multiple data centers according to the grouping size corresponding to each of the multiple parallel computing strategies, resulting in a unit group corresponding to the parallel computing strategy. The parallel computing strategy is used to partition the model to be trained and / or the training sample set for distributed training of the model. The model to be trained is generally a large language model, such as GPT (Generative Pre-trained Transformer), Megatron-LM (Language Model), LLama, etc., which have a large number of parameters.

[0033] Multiple parallel computing strategies can be at least two parallel computing strategies from a set of parallel computing strategies. The set of parallel computing strategies includes, but is not limited to, DP (Data Parallelism), PP (Pipeline Parallelism), TP (Tensor Parallelism), EP (Expert Parallelism), and variations or combinations of the above parallel computing strategies.

[0034] Data parallelism is a strategy that involves splitting a dataset into multiple subsets, each assigned to a cluster of computing units (or computing units). The cluster of computing units trains the model replica it deploys by receiving the subset. Data parallelism mainly focuses on the division of the dataset and the synchronization of model replicas. In data parallelism, each cluster of computing units maintains a complete copy of the model and independently calculates the gradient for its assigned batch. Gradient synchronization is a key step in data parallelism, which can be achieved through parameter servers or decentralized AllReduce algorithms.

[0035] Pipeline parallelism is a way of model parallelism that involves splitting different layers of the model and assigning them to different clusters of computing units (or computing units), thereby implementing a pipeline-style computing process. Pipeline parallelism mainly focuses on the computing process between model layers and is a division of different Transformer layers in the model. In pipeline parallelism, each computing unit is responsible for a part of the model layers, and data flows between devices by layer for forward and backward propagation. Pipeline parallelism can improve device utilization and reduce device idle time through the MicroBatch pipeline parallelism scheme.

[0036] Tensor parallelism is a strategy that involves splitting tensors in the model and assigning different tensor parts to different clusters of computing units (or computing units), thereby implementing parallel computing. Tensor parallelism mainly focuses on tensor operations within the model and is a division of specific matrix operations within the layer. It needs to solve the problem of how to split parameters to different devices and how to ensure mathematical consistency after splitting. In tensor parallelism, each cluster of computing units (or computing units) usually only saves a part of the model parameters and exchanges data with other devices through communication links.

[0037] In expert parallelism, the model is divided into multiple "experts" or sub-models, each responsible for processing a part of the input or output of the model or performing a specific task in the model. These experts are assigned to different computing devices to achieve parallel computing. Through expert parallelism, the memory requirements of a single computing device can be further reduced, the memory efficiency can be improved, and the training process can be accelerated.

[0038] Multiple data centers refer to the number of data centers whose data is not less than two. Multiple data centers generally refer to data centers that have a certain interval distance in physical space, but it is not excluded that multiple data centers do not have an interval distance in physical space. Each data center includes multiple computing units, and in general, each data center includes a large number of computing units. Computing units are, for example, GPUs, CPUs, etc.

[0039] As an example, the execution subject can determine, based on the received group size indication instruction, a group size corresponding to each of the plurality of parallel computing strategies; and for each of the plurality of parallel computing strategies, group the computing units in the plurality of data centers according to the group size corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy.

[0040] As another example, the execution subject can determine, by a group size determination model, a group size corresponding to each of the plurality of parallel computing strategies according to characteristics (such as model structure, parameter size, etc.) of the to-be-trained model and the number of computing units of the plurality of data centers; and for each of the plurality of parallel computing strategies, group the computing units in the plurality of data centers according to the group size corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy.

[0041] For example, the plurality of data centers includes data centers DC1 and DC2, and the number of computing units in each data center is 256; the plurality of parallel computing strategies includes strategies P1, P2 and P3, and the group sizes corresponding to the strategies P1, P2 and P3 are 4, 8 and 32 respectively; then the 512 computing units in the data centers DC1 and DC2 are divided into 128 (512 ÷ 4) unit groups corresponding to the strategy P1, 64 (512 ÷ 8) unit groups corresponding to the strategy P2, and 16 (512 ÷ 32) unit groups corresponding to the strategy P3.

[0042] In some optional implementations of the embodiment, the plurality of parallel computing strategies includes a data parallel strategy, a pipeline parallel strategy and a tensor parallel strategy. For example, the Megatron splitting framework of PTD (Parallel Task Decomposition) parallel is used in the implementation.

[0043] In the implementation, the execution subject can determine the group size corresponding to each of the plurality of parallel computing strategies in the following manner:

[0044] For the tensor parallel strategy, the group size corresponding to the tensor parallel strategy is determined according to the computing architecture of the service device where the computing unit is located.

[0045] Among the three parallel computing strategies, the tensor parallel strategy produces the largest scale of communication data, and requires high-speed and low-latency communication interaction between multiple GPUs in the same unit group. Taking the Nvlink architecture commonly used in the industry as an example, it is a high-speed and low-latency inter-GPU communication interface that allows direct data exchange between GPUs without passing through the CPU or other intermediate devices, thereby enhancing communication between GPUs, improving performance, and enabling seamless parallel processing. Limited by the Nvlink architecture, fast communication can be achieved between the 8 GPUs in the server. Therefore, the grouping scale corresponding to the tensor parallel strategy can be determined as 8.

[0046] It should be noted that based on technological development, the computing architecture and communication architecture of the service device will be iteratively updated. Based on this, the grouping scale corresponding to the tensor parallel strategy can be updated following the iterative update of the computing architecture and communication structure of the service device.

[0047] For the data parallel strategy and the pipeline parallel strategy, the grouping scale corresponding to the data parallel strategy is greater than the grouping scale corresponding to the pipeline parallel strategy. The grouping scale corresponding to the data parallel strategy and the pipeline parallel strategy is determined in this way.

[0048] With reference to Figure 3 , a schematic diagram of the computing performance of the data parallel strategy and the pipeline parallel strategy under different grouping scales is shown.

[0049] When the batch size is 512, as the grouping scale corresponding to the pipeline parallel strategy increases, the grouping scale corresponding to the data parallel strategy decreases, and the computing performance of each GPU decreases. teraFLOP / s represents trillion floating point operations per second.

[0050] When the batch size is 32, as the grouping scale corresponding to the pipeline parallel strategy increases, the grouping scale corresponding to the data parallel strategy decreases, and the computing performance of each GPU also decreases.

[0051] Therefore, in the present implementation, the grouping scale corresponding to the data parallel strategy can be greater than the grouping scale corresponding to the pipeline parallel strategy. The grouping scale corresponding to the data parallel strategy and the pipeline parallel strategy is determined in this way.

[0052] As an example, two data centers each include 8192 GPUs, and the grouping sizes corresponding to the data parallel strategy, the pipeline parallel strategy, and the tensor parallel strategy are 128, 16, and 8 respectively. That is, for each data center, the 8192 GPUs are divided into 64 groups corresponding to the data parallel strategy, 512 groups corresponding to the pipeline parallel strategy, and 1024 groups corresponding to the tensor parallel strategy. Among them, the groups corresponding to the data parallel strategy include 16 groups corresponding to the tensor parallel strategy and 8 groups corresponding to the pipeline parallel strategy.

[0053] In the present implementation, the specific determination manner of the grouping sizes corresponding to the plurality of parallel computing strategies is provided, which helps to further improve the rationality of grouping, and further improve the communication efficiency and rate in the distributed training process.

[0054] In step 202, a target parallel computing strategy is determined from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration.

[0055] In the present embodiment, the execution subject can determine a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay duration.

[0056] The communication data generated by the parallel computing strategy refers to the data that needs to be synchronized between the computing units when running the parallel computing strategy. Specifically, for the data parallel strategy, the generated communication data mainly involves model parameters or gradient data that need to be synchronized by the GPU to ensure that different model copies of the model to be trained are consistent. For the pipeline parallel strategy, the generated communication data refers to the data transmitted between different stages of the pipeline. For example, when a GPU completes the computation of the part it is responsible for, it needs to pass the result to the next GPU for subsequent computation. For the tensor parallel strategy, the generated communication data is the data related to model weights or computation graphs transmitted between GPUs. Specifically, it can include model parameters or gradient information that need to be synchronized or exchanged between GPUs during the training process.

[0057] The higher the sensitivity of the communication data generated by the parallel computing strategy to the communication delay duration, the more acceptable the communication data generated by the parallel computing strategy is to the communication delay, and the smaller the negative impact (e.g., reduced training efficiency) on the distributed training process. The higher the sensitivity of the communication data generated by the parallel computing strategy to the communication delay duration, the less acceptable the communication data generated by the parallel computing strategy is to the communication delay, and the greater the negative impact (e.g., reduced training efficiency) on the distributed training process.

[0058] As an example, the above execution subject or an electronic device in communication connection with the above execution subject is provided with correspondence relationship data representing the correspondence relationship between the parallel computing strategy and the sensitivity; after determining the parallel computing strategy adopted by the distributed training process of the to-be-trained model, the sensitivity of the parallel computing strategy can be determined through the correspondence relationship data; and the parallel computing strategy with the lowest sensitivity among the multiple parallel computing strategies is determined as the target parallel computing strategy. Wherein, in the historical distributed training process across data centers, the correspondence relationship data is determined through the analysis of the communication efficiency of different parallel running strategies.

[0059] As another example, the above execution subject can sort the sensitivities of the multiple parallel computing strategies through a sensitivity sorting model, so as to determine the parallel computing strategy with the lowest sensitivity among the multiple parallel computing strategies as the target parallel computing strategy.

[0060] In some optional implementation manners of the embodiment, the above execution subject can execute the above step 202 in the following manner: in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivities corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively, the data parallel strategy is determined as the target parallel computing strategy.

[0061] The sensitivity corresponding to the data parallel strategy being lower than the sensitivities corresponding to the tensor parallel strategy and the pipeline parallel strategy respectively is mainly based on the following two points:

[0062] 1. The message size of the communication data generated by the pipeline parallel strategy is smaller, and the message size of the communication data generated by the data parallel strategy is larger, and the communication data with large message size is more insensitive to long-distance communication across data centers.

[0063] The pipeline parallel strategy allocates different layers or stages of the model to different computing units for calculation, and data is transmitted between stages in a pipeline manner. Therefore, the communication under the pipeline parallel strategy mainly occurs between adjacent computing units, and the output of the previous layer (as the input of the next layer) and possible gradient information are transmitted. Because only data needs to be transmitted between adjacent devices under the pipeline parallel strategy, rather than broadcasted in the entire cluster, the communication volume is relatively small. In addition, since the output of each layer or stage usually only contains the calculation result of the layer or stage, the size of these data is also relatively limited.

[0064] The data parallel strategy replicates the entire model to multiple clusters of computing units (or computing units), each of which independently processes a portion of the data set and calculates the gradient. At the end of each round of training, the gradient information needs to be exchanged between the various clusters of computing units for synchronous updating. Because the data parallel strategy requires a large communication at the end of each round to synchronize the gradient information of all device clusters. These gradient information involves all the parameters of the model, so the communication volume is usually large. In addition, as the model size increases and the number of parameters increases, the size of the gradient information will also increase accordingly.

[0065] 2. In the data parallel processing process under the data parallel strategy, the communication (data transmission or synchronization) process and the calculation process are effectively overlapped or executed in parallel, which can maximize resource utilization, reduce overall calculation time, and improve the efficiency of parallel computing.

[0066] In data parallelism, the model is replicated to multiple clusters of computing units, each of which processes a subset of input data. Ideally, when a certain cluster of computing units is performing calculations, other clusters of computing units can simultaneously perform data transmission or prepare data for the next calculation task. This overlap can significantly reduce the total calculation time, because communication and calculation are no longer serial, but can be performed in parallel. Therefore, the overlap of the communication process and the calculation process in the data parallel process can to some extent mask the problem of increased network latency.

[0067] In the present implementation, for the mixed parallel scene of the data parallel strategy, the pipeline parallel strategy and the tensor parallel strategy, the data parallel strategy is determined as the target parallel strategy, which helps to further reduce the influence of the communication delay between data centers on the training process.

[0068] Step 203, in the distributed training process of the to-be-trained model, control the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between multiple data centers, and control the unit group corresponding to other parallel computing strategies to interact the communication data generated by other parallel computing strategies within the corresponding data center.

[0069] In the present embodiment, the above execution subject can control the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between multiple data centers, and control the unit group corresponding to other parallel computing strategies to interact the communication data generated by other parallel computing strategies within the corresponding data center in the distributed training process of the to-be-trained model.

[0070] Specifically, for the plurality of unit groups corresponding to the target parallel computing strategy, data synchronization needs to be performed between the plurality of unit groups, and therefore, communication data generated by the target parallel computing strategy needs to be interacted between the plurality of data centers; for the unit groups corresponding to other parallel computing strategies, data communication needs to be performed only within the unit groups, and therefore, communication data generated by the target parallel computing strategy needs to be interacted only within the data center.

[0071] In some optional implementations of the embodiment, the execution subject can perform the process of "controlling the unit groups corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between the plurality of data centers" in step 203 in the following manner:

[0072] First, a plurality of first ring communication links between the plurality of unit groups corresponding to the target parallel computing strategy are established.

[0073] In the first ring communication link, one computing unit in each unit group of the plurality of unit groups is included, and the plurality of computing units in the first ring communication link include the same part of the to-be-trained model.

[0074] Second, in the first ring communication link, the communication data generated by the target parallel computing strategy is interacted in a global reduction communication manner.

[0075] Continuing to refer to Figure 4 , a grouping diagram of the data center is shown.

[0076] In Figure 4 , two data centers DC1 and DC2 are included, each data center includes 8192 GPUs, and the grouping scales corresponding to the data parallel strategy, the pipeline parallel strategy, and the tensor parallel strategy are 128, 16, and 8 respectively. The two data centers include 128 unit groups corresponding to the data parallel strategy, respectively DP 1-DP 128. Each DP group includes 16 unit groups corresponding to the tensor parallel strategy, respectively TP 1-TP 16, each TP group can be a computing node (for example, a server), and each TP group includes 1-8 computing units in total, that is, 8 computing units. Each DP group includes 8 unit groups corresponding to the pipeline parallel strategy, respectively PP 1-PP 8.

[0077] In each DP group, the complete parameters of one to-be-trained model are included, and in each computing unit in one DP group, the parameters of a part of the model in the to-be-trained model are included.

[0078] The first ring communication link is, for example, Figure 4 the ring communication link represented by the solid line and the ring communication link represented by the dashed line in Figure 4As shown in the grouping manner, it can be known that 128 first ring communication links are included in total. The thick solid line part and the thin dotted line part in the figure represent the communication between the two data centers.

[0079] The global reduction communication manner is an All Reduce communication manner, and specifically is a Ring All Reduce communication manner. The Ring All Reduce is divided into three steps: Split, ScatterReduce, and AllGather.

[0080] 1. The Split stage: According to the size N of the cluster, the data to be synchronized is evenly divided into N data blocks.

[0081] 2. The ScatterReduce stage: N-1 times of ScatterReduce iteration is performed. In each iteration, the GPU sends a data block to its right neighbor and receives a data block from its left neighbor and accumulates the received data block to the local data block corresponding to the received data block. The nodes exchange data in turn, so that each node only contains a part (1 / N) of the final result. The block sent and received by each GPU is different in each iteration; the nth GPU starts from sending block N and receiving block N-1, and in the subsequent iteration process, it sends the data block accumulated based on the previous iteration each time. When the ScatterReduce is completed, each node has a data block, which accumulates the data of the corresponding data block of all other worker nodes.

[0082] 3. The AllGather stage: The nodes exchange data again to obtain the complete result. This process is the same as scatter-reduce (N-1 times of sending and receiving iteration), except that the value received by the GPU is not accumulated, but simply covers the block. The nth GPU first sends the (n+1)th block and receives the nth block, and then always sends the data block it just received in the subsequent iterations.

[0083] After the AllGather operation is completed, all data blocks of all worker nodes contain data from other worker nodes.

[0084] In the implementation mode, a specific communication manner of a unit group corresponding to a target parallel computing strategy is provided, which, in combination with the ring communication link and the global reduction communication manner, helps to further reduce the influence of the communication delay between data centers on the training process.

[0085] In some optional implementation modes of the embodiment, the execution subject can execute the second step in the following manner:

[0086] Firstly, a target transmission rate of a communication channel between the plurality of data centers is determined according to a transmission rate between the computing units in the first ring communication link and a preset redundancy multiple.

[0087] The preset redundancy multiple can be set according to actual conditions, for example, the preset redundancy multiple is 2. As an example, the target transmission rate is at least the preset redundancy multiple of the transmission rate between the computing units in the first ring communication link.

[0088] For example, if the transmission rate of a remote network interface card (RNIC) of the computing unit is 400G, then 800G of bandwidth can be configured across the data centers in each first ring communication link. This is because, under the condition that the computing node meets the network affinity, there is only one pair of communication traffic across the data centers in each first ring communication link, and therefore 400G x 2 of bandwidth is required to support high-throughput communication of a single first ring communication link. In this way, the interconnection bandwidth of 128 first ring communication links between DC1 and DC2 will reach 102.4T (800G x 128).

[0089] Then, in the first ring communication link, the communication data generated by the target parallel computing strategy is interacted using a global reduction communication mode and the target transmission rate.

[0090] In the present implementation, a higher transmission rate is set for the communication channel between the plurality of data centers, which can avoid the problem of channel congestion and improve the communication efficiency of the communication data across the data centers.

[0091] In some optional implementations of the present embodiment, the above-mentioned execution subject can execute the above-mentioned process of “interacting the communication data generated by the target parallel computing strategy using the target transmission rate” in the following manner:

[0092] Firstly, a target capacity of a buffer corresponding to the communication channel between the plurality of data centers is set based on a preset capacity threshold.

[0093] The preset capacity threshold can be set according to actual conditions, which is not limited herein.

[0094] In the communication process based on NCCL (NVIDIA Collective Communications Library), NCCL_BUFFSIZE determines the size of the buffer on each communication channel, thereby indirectly determining the slice size of data transmission. Since data transmission needs to pass through the buffer, if the buffer is too small, the read and write of the buffer are more likely to be blocked. Especially in the process of long-distance cross-data center communication, the network delay increases. At this time, if a traditional small-capacity buffer is used, the whole slice of a single chunk waits for the receiver to confirm after sending, and the ideal multi-slice pipeline transmission mode degenerates into the mode of single chunk sending, waiting for confirmation and then sending again, resulting in reduced network transmission performance.

[0095] Through experiments, it is found that the communication based on a large-capacity (not less than a preset capacity threshold) buffer can effectively improve the performance of NCCL All Reduce in long-distance. For example, in the case of cross-data center communication within 100 kilometers, the use of a large-capacity buffer can obtain more than 200% performance improvement compared with the default capacity (4M) buffer.

[0096] Of course, simply increasing the capacity of the buffer will occupy more GPU memory. How to balance the relationship between the buffer capacity and the GPU memory needs to be judged according to the actual situation, but the test results show that in the training of long-distance communication between data centers, the large-capacity buffer communication of NCCL is a direction that can effectively optimize network communication and reduce performance loss.

[0097] Then, the communication data generated by the target parallel computing strategy is interacted by using the target transmission rate and the buffer with the target capacity.

[0098] In the present embodiment, by increasing the capacity of the buffer and combining the increased target transmission rate, the problem of channel congestion can be further avoided, and the communication efficiency of the communication data between the data centers can be improved.

[0099] In some optional implementations of the present embodiment, the execution subject can execute the process of "controlling the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between the multiple data centers" in step 203 in the following manner:

[0100] For the tensor parallel strategy:

[0101] First, a second ring communication link is established based on the multiple computing units in the unit group corresponding to the tensor parallel strategy. Then, in the second ring communication link, the communication data generated by the tensor parallel strategy is interacted by using a global reduction communication mode.

[0102] With reference to the foregoing Figure 4 For each of the 1-8 computing units in the TP group, a second ring communication link is established; and then a global reduction communication mode is adopted to exchange data within the TP group.

[0103] For the pipeline parallel strategy:

[0104] Between the plurality of computing units in the unit group corresponding to the pipeline parallel strategy, a point-to-point communication mode is adopted to exchange communication data generated by the pipeline parallel strategy.

[0105] The point-to-point communication allows two specific GPUs to directly exchange data.

[0106] With reference to the foregoing Figure 4 For each of the 16 computing units in the PP group, in the forward calculation process, the current computing unit receives data from the previous computing unit, processes it, and transmits the processed data to the next computing unit; and in the reverse calculation process, the transmission direction is reversed, thereby realizing data communication between the computing units in the same PP.

[0107] In the implementation mode, a specific communication mode of the unit group corresponding to other parallel computing strategies is provided, and different parallel computing strategies adopt their suitable communication modes, thereby improving the data communication efficiency within the unit group corresponding to other parallel computing strategies.

[0108] With reference to the foregoing Figure 5 , Figure 5 is an example of an application scenario 500 of the cross-data-center distributed training method according to the embodiment. The server 501 needs to control the computing units in the two data centers 502 and 503 to perform distributed training on a to-be-trained model. First, for each of a plurality of parallel computing strategies, the computing units in the plurality of data centers are grouped according to the grouping scale corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to split the to-be-trained model and / or a training sample set to train the to-be-trained model in a distributed manner; then, according to the sensitivity of the communication data generated by the plurality of parallel computing strategies to the communication delay time length, a target parallel computing strategy is determined from the plurality of parallel computing strategies; finally, in the process of distributed training of the to-be-trained model, the unit group corresponding to the target parallel computing strategy is controlled to exchange the communication data generated by the target parallel computing strategy between the plurality of data centers, and the unit groups corresponding to other parallel computing strategies are controlled to exchange the communication data generated by other parallel computing strategies within the corresponding data centers.

[0109] In the embodiment, a distributed training method across data centers is provided, which divides the computing units in the plurality of data centers based on the grouping scale corresponding to each of the plurality of parallel computing strategies, improves the rationality of the unit groups, and helps to improve the data communication efficiency based on the unit groups; based on the sensitivity of the communication data of the plurality of parallel computing strategies to the communication delay time length, the sensitive communication data is controlled to interact across data centers, and the non-sensitive communication data is controlled to interact within the data center, thereby reducing the influence of the communication delay between the data centers on the training process. Thus, based on the plurality of data centers and the plurality of parallel computing strategies, the training efficiency of the distributed training process is improved while maintaining the communication efficiency.

[0110] In some optional implementation manners of the embodiment, the execution subject can further perform the following operation: for the target computing unit in the unit group corresponding to the target parallel computing strategy, which performs data interaction between the plurality of data centers, the adaptive routing function of the target computing unit is closed.

[0111] After the AR is enabled, the processing mechanism is to use the bitmap method because the AR needs to process out-of-order reception. Therefore, a relatively small sending window is needed. If the sending window is too large, the processing complexity will be increased. Therefore, the network card reduces the size of the sending window (which is a speed reduction behavior itself). After the sending window is reduced, the frequency of waiting for the ACK message will be increased. Because the RTT (Round-Trip Time) delay is increased in the long-distance communication across data centers, the overall superposition further reduces the sending rate of the message.

[0112] It is found through experiments that the number of messages that need to be confirmed increases sharply after the AR is enabled. For example, when 10,000 messages are captured, the number of messages that need to be confirmed is 676 after the AR is enabled, and the number of messages that need to be confirmed is 46 after the AR is disabled, and the number of messages decreases sharply.

[0113] Under long-distance RDMA (Remote Direct Memory Access), if AR is enabled, the following situations may occur: (1) The packet needs to carry an additional RETH (Remote Extended Transport Header) extension header, resulting in a smaller payload and reduced efficiency; (2) The packet receiver needs to process the received out-of-order packets, so the sending window size is reduced to facilitate the delivery of out-of-order packets, thereby reducing the sending rate of the sending end; (3) The number of packets that need to be acknowledged increases; (4) The long-distance RTT increases, the number of packets that need to be ACKed increases, further lengthening the packet acknowledgment waiting time, and reducing the sending efficiency.

[0114] In the present implementation, by turning off the adaptive routing function of the target computing unit, the above situations can be effectively avoided, and the communication efficiency of the communication data can be improved.

[0115] In some optional implementations of the present embodiment, the execution subject can further perform the following operations: for a preset collective communication library on which a communication process between a plurality of data centers is based, turning off an adaptive routing function of the preset collective communication library.

[0116] The preset collective communication library is, for example, NCCL, which utilizes RDMA technology, an efficient data transfer method that allows computers in a network to directly access each other's memory without the intervention of the operating system or CPU. Through RDMA, NCCL can achieve low-latency, high-bandwidth data transfer. NCCL supports both point-to-point communication and cluster communication modes. Point-to-point communication allows direct data exchange between two specific GPUs, while cluster communication involves data exchange between multiple GPUs. NCCL selects the most suitable communication mode according to specific communication needs to achieve efficient data transfer. NCCL is optimized for NVIDIA GPUs and networks, and can achieve high bandwidth and low latency on high-speed interconnection networks such as PCIe (Peripheral Component Interconnect express), NVLink, etc. This hardware acceleration makes NCCL perform well in scenarios such as deep learning training that require a lot of GPU-to-GPU communication. When configuring the NCCL library, turning off AR can further improve the communication efficiency of the communication data.

[0117] In some optional implementations of the present embodiment, the execution subject can further perform the following operations:

[0118] First, determine the performance degradation ratio information of the communication data generated by the target parallel computing strategy in the long-distance communication process between multiple data centers, compared to the short-distance communication process within one data center.

[0119] In the present implementation, the above execution subject can use the performance test tool nccl_test in the NCCL library to measure the communication performance under different network conditions (such as different distances and different bandwidths), so as to evaluate the influence of network delay and bandwidth on training performance.

[0120] For example, the performance degradation ratio information is 316% when the performance of the short-distance communication process within one data center is taken as 100%.

[0121] Then, determine the proportion information of the time length of the part of the communication process that is not covered by the computation process in the communication data generated by the target parallel computing strategy in one iteration training cycle.

[0122] Continuing to refer to Figures 6A-6C , Figure 6A , a communication data change diagram of a complete iteration cycle is shown, Figure 6B a communication diagram of the communication data generated by the data parallel strategy is shown, Figure 6C and a communication diagram of the communication data generated by the pipeline parallel strategy is shown.

[0123] As shown in Figure 6A , in a complete iteration cycle, only the communication data corresponding to the PP strategy is used in the forward computation, and the communication data of the DP strategy and the PP strategy is used in the backward computation. There is a short period of DP strategy communication data that is not covered by the computation after the backward computation is completed, which fully proves that in the DP strategy, most of the communication process overlaps with the computation process. The communication data of the DP strategy is much higher than that of the PP strategy, which also fully illustrates that the message size of the communication data of the DP strategy is much larger than that of the PP strategy. The highest flow instant of the curve is the communication data flow of the communication process that is not covered by the computation, which is also the DP communication part that affects the overall performance of the distributed training process in the fast data center.

[0124] As shown in Figure 6B , 6C , the circled DP communication part corresponds to no PP flow, and the proportion of the DP part that is not covered by the computation in the entire iteration cycle is about 5% (conservative) in the above figure.

[0125] Finally, according to the performance degradation ratio information and the proportion information, the performance loss information of the distributed training process is determined.

[0126] Continuing with the example where the performance degradation ratio information is 316% and the proportion information is 5%, the performance loss information is 316%*5%+95%-1=10.08%.

[0127] In the present implementation, a performance estimation method in a cross-data center communication scenario is provided, which can quantify the performance loss in the distributed training process and provide evaluation accuracy of the distributed training process.

[0128] Continuing to refer to Figure 7 , a schematic flow 700 of yet another embodiment of a cross-data center distributed training method according to the present disclosure is shown. In flow 700, the following steps are included:

[0129] Step 701, according to the computing architecture of the service device where the computing unit in the plurality of data centers is located, determine the grouping scale corresponding to the tensor parallel strategy.

[0130] Step 702, using the determination method that the grouping scale corresponding to the data parallel strategy is greater than the grouping scale corresponding to the pipeline parallel strategy, determine the grouping scale corresponding to each of the data parallel strategy and the pipeline parallel strategy.

[0131] Step 703, for each of the data parallel strategy, the pipeline parallel strategy and the tensor parallel strategy, according to the grouping scale corresponding to the parallel computing strategy, group the computing units in the plurality of data centers to obtain the unit group corresponding to the parallel computing strategy.

[0132] Wherein, the parallel computing strategy is used to split the to-be-trained model and / or the training sample set to distribute the training of the to-be-trained model.

[0133] Step 704, in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivity corresponding to each of the tensor parallel strategy and the pipeline parallel strategy, determine the data parallel strategy as the target parallel computing strategy.

[0134] Step 705, based on the plurality of computing units in the unit group corresponding to the tensor parallel strategy, establish a second ring communication link.

[0135] Step 706, in the second ring communication link, use a global reduction communication mode to interact with the communication data generated by the tensor parallel strategy.

[0136] Step 707, between the plurality of computing units in the unit group corresponding to the pipeline parallel strategy, use a point-to-point communication mode to interact with the communication data generated by the pipeline parallel strategy.

[0137] Step 708, establish a plurality of first ring communication links between the plurality of unit groups corresponding to the data parallel computing strategy.

[0138] The first ring-shaped communication link includes one computing unit in each of a plurality of unit groups, and the plurality of computing units in the first ring-shaped communication link include the same part of the to-be-trained model.

[0139] At step 709, in the first ring-shaped communication link, the communication data generated by the interactive data parallel computing strategy is exchanged in a global reduction communication manner.

[0140] The flow 700 of the cross-data-center distributed training method in this embodiment specifically illustrates the determination process of the grouping scale of the parallel computing strategy, the determination process of the target parallel computing strategy, and the communication process of each parallel computing strategy. On the basis of maintaining communication efficiency, the training efficiency of the distributed training process is improved based on multiple data centers and multiple parallel computing strategies.

[0141] Continuing to refer to Figure 8 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a cross-data-center distributed training device. The system embodiment corresponds to the method embodiment shown in Figure 2 , and the system can be specifically applied to various electronic devices.

[0142] As shown in Figure 8 , the cross-data-center distributed training device 800 includes a grouping unit 801 configured to, for each of a plurality of parallel computing strategies, group computing units in a plurality of data centers according to a grouping scale corresponding to the parallel computing strategy, to obtain a unit group corresponding to the parallel computing strategy, wherein the parallel computing strategy is used to split a to-be-trained model and / or a training sample set to train the to-be-trained model in a distributed manner; a strategy determination unit 802 configured to determine a target parallel computing strategy from the plurality of parallel computing strategies according to the sensitivity of the communication data generated by each of the plurality of parallel computing strategies to the communication delay time length; and a communication control unit 803 configured to, in a distributed training process of the to-be-trained model, control the unit group corresponding to the target parallel computing strategy to exchange the communication data generated by the target parallel computing strategy between the plurality of data centers, and control the unit groups corresponding to other parallel computing strategies to exchange the communication data generated by the other parallel computing strategies within the corresponding data centers.

[0143] In some optional implementations of this embodiment, the plurality of parallel computing strategies includes a data parallel strategy, a pipeline parallel strategy, and a tensor parallel strategy, and the device further includes a scale determination unit (not shown in the figure) configured to: determine the grouping scale corresponding to the tensor parallel strategy according to the computing architecture of a service device where the computing unit is located; and determine the grouping scales corresponding to the data parallel strategy and the pipeline parallel strategy in a determination manner that the grouping scale corresponding to the data parallel strategy is greater than the grouping scale corresponding to the pipeline parallel strategy.

[0144] In some optional implementation of the embodiment, the policy determination unit 802 is further configured to: in response to the sensitivity corresponding to the data parallel strategy being lower than the sensitivity corresponding to each of the tensor parallel strategy and the pipeline parallel strategy, determine the data parallel strategy as the target parallel computing strategy.

[0145] In some optional implementation of the embodiment, the communication control unit 803 is further configured to: establish a plurality of first ring communication links between a plurality of cell groups corresponding to the target parallel computing strategy, wherein each of the plurality of first ring communication links comprises one computing cell in each of the plurality of cell groups, and the plurality of computing cells included in the first ring communication link comprise the same part of the model to be trained; and in the first ring communication link, interact with the communication data generated by the target parallel computing strategy in a global reduction communication mode.

[0146] In some optional implementation of the embodiment, the communication control unit 803 is further configured to: determine a target transmission rate of a communication channel between the plurality of data centers according to a transmission rate between the computing cells in the first ring communication link and a preset redundancy multiple; and in the first ring communication link, interact with the communication data generated by the target parallel computing strategy in a global reduction communication mode and the target transmission rate.

[0147] In some optional implementation of the embodiment, the communication control unit 803 is further configured to: set a target capacity of a buffer corresponding to the communication channel between the plurality of data centers based on a preset capacity threshold; and interact with the communication data generated by the target parallel computing strategy using the buffer with the target transmission rate and the target capacity.

[0148] In some optional implementation of the embodiment, the communication control unit 803 is further configured to: establish a second ring communication link based on a plurality of computing cells in a cell group corresponding to the tensor parallel strategy; in the second ring communication link, interact with the communication data generated by the tensor parallel strategy in a global reduction communication mode; and between a plurality of computing cells in a cell group corresponding to the pipeline parallel strategy, interact with the communication data generated by the pipeline parallel strategy in a point-to-point communication mode.

[0149] In some optional implementation of the embodiment, the apparatus further comprises a first setting unit (not shown in the figure) configured to, for a target computing cell in a cell group corresponding to the target parallel computing strategy, which interacts with data between a plurality of data centers, turn off an adaptive routing function of the target computing cell.

[0150] In some optional implementations of the embodiment, the device further includes a second setting unit (not shown in the figure) configured to, for the preset collective communication library according to which the communication process between the plurality of data centers is performed, turn off an adaptive routing function of the preset collective communication library.

[0151] In some optional implementations of the embodiment, the device targets an overlap of a computation process of communication data corresponding to a parallel computing strategy and a communication process, and the device further includes a loss determination unit (not shown in the figure) configured to: determine performance degradation proportion information of the communication data generated by the target parallel computing strategy in a long-distance communication process between the plurality of data centers compared to a short-distance communication process within one data center; determine proportion information of a time length of a part of the communication process that is not covered by the computation process in one iteration training period in the communication data generated by the target parallel computing strategy; and determine performance loss information of the distributed training process according to the performance degradation proportion information and the proportion information.

[0152] In the embodiment, a distributed training device across data centers is provided, which divides the computing units in the plurality of data centers based on the grouping sizes corresponding to the plurality of parallel computing strategies respectively, improves the rationality of the unit groups, and helps to improve the data communication efficiency based on the unit groups; controls the sensitive communication data to interact across data centers and the non-sensitive communication data to interact within data centers based on the sensitivity of the communication data of the plurality of parallel computing strategies to the communication delay time length, and reduces the influence of the communication delay between data centers on the training process. Thus, on the basis of maintaining the communication efficiency, the training efficiency of the distributed training process is improved based on the plurality of data centers and the plurality of parallel computing strategies.

[0153] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, which includes at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the distributed training method across data centers described in any of the embodiments.

[0154] According to the embodiments of the present disclosure, the present disclosure further provides a readable storage medium storing computer instructions for enabling a computer to implement the distributed training method across data centers described in any of the embodiments when the computer executes the computer instructions.

[0155] The present disclosure provides a computer program product, which enables the distributed training method across data centers described in any of the embodiments when the computer program is executed by a processor.

[0156] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0157] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0158] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0159] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the distributed training method across data centers. For example, in some embodiments, the distributed training method across data centers can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the distributed training method across data centers described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured, by way of firmware or otherwise, to execute the distributed training method across data centers.

[0160] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0161] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable computing apparatuses to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0164] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0165] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically occurring over the network. The relationship can be created by the use of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services; it can also be a server of a distributed system, or a server combined with a blockchain.

[0166] According to the technical scheme of the embodiment of the present disclosure, a distributed training method and device across data centers are provided, the computing units in the plurality of data centers are divided based on the grouping scales corresponding to the plurality of parallel computing strategies respectively, the rationality of the unit groups is improved, which helps to improve the data communication efficiency based on the unit groups; based on the sensitivity of the communication data of the plurality of parallel computing strategies to the communication delay time length, the sensitive communication data is controlled to interact across data centers, the non-sensitive communication data is controlled to interact within the data centers, and the influence of the communication delay between the data centers on the training process is reduced. Thus, on the basis of maintaining the communication efficiency, based on the plurality of data centers and the plurality of parallel computing strategies, the training efficiency of the distributed training process is improved.

[0167] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical scheme provided by the present disclosure can be achieved, which is not limited herein.

[0168] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A distributed training method across data centers, comprising: For each of the multiple parallel computing strategies, the computing units in the multiple data centers are grouped according to the grouping size corresponding to the parallel computing strategy to obtain the unit group corresponding to the parallel computing strategy. The parallel computing strategy is used to split the model to be trained and / or the training sample set for distributed training of the model to be trained. Based on the sensitivity of the communication data generated by each of the multiple parallel computing strategies to the communication latency, a target parallel computing strategy is determined from the multiple parallel computing strategies. During the distributed training of the model to be trained, the unit group corresponding to the target parallel computing strategy is controlled to interact with the communication data generated by the target parallel computing strategy between multiple data centers, and the unit group corresponding to other parallel computing strategies is controlled to interact with the communication data generated by other parallel computing strategies within its respective data center.

2. The method according to claim 1, wherein, The various parallel computing strategies mentioned include data parallelism, pipelined parallelism, and tensor parallelism. The method further includes: The group size corresponding to the tensor parallel strategy is determined based on the computing architecture of the service device where the computing unit is located. The method for determining that the group size corresponding to the data parallel strategy is greater than the group size corresponding to the pipeline parallel strategy determines the group size corresponding to each of the data parallel strategy and the pipeline parallel strategy.

3. The method according to claim 2, wherein, The step of determining a target parallel computing strategy from the multiple parallel computing strategies based on the sensitivity of communication data generated by each of the multiple parallel computing strategies to communication latency includes: Since the sensitivity of the data parallel strategy is lower than that of the tensor parallel strategy and the pipeline parallel strategy, the data parallel strategy is determined as the target parallel computing strategy.

4. The method according to any one of claims 1-3, wherein, The unit group controlling the target parallel computing strategy interacts with the communication data generated by the target parallel computing strategy among multiple data centers, including: Establish multiple first ring communication links between multiple unit groups corresponding to the target parallel computing strategy, wherein each first ring communication link includes a computing unit in each of the multiple unit groups, and the multiple computing units in the first ring communication link include some of the same training models. In the first ring communication link, a global reduction communication method is used to exchange the communication data generated by the target parallel computing strategy.

5. The method according to claim 4, wherein, In the first ring communication link, a global reduction communication method is used to exchange communication data generated by the target parallel computing strategy, including: The target transmission rate of the communication channel between the multiple data centers is determined based on the transmission rate between the computing units in the first ring communication link and the preset redundancy multiple. In the first ring communication link, a global reduction communication method and the target transmission rate are used to exchange communication data generated by the target parallel computing strategy.

6. The method according to claim 5, wherein, Using the target transmission rate, the communication data generated by the target parallel computing strategy includes: Based on a preset capacity threshold, the target capacity of the buffer corresponding to the communication channel between the multiple data centers is set; The target transmission rate and the target capacity buffer are used to exchange the communication data generated by the target parallel computing strategy.

7. The method according to claim 3, wherein, The unit groups that control other parallel computing strategies interact with the communication data generated by other parallel computing strategies within their respective data centers, including: A second ring communication link is established based on multiple computing units in the unit group corresponding to the tensor parallel strategy. In the second ring communication link, a global reduction communication method is used, and communication data generated by an interactive tensor parallel strategy is employed; and In the unit group corresponding to the pipelined parallel strategy, multiple computing units in the unit group use a point-to-point communication method to exchange the communication data generated by the pipelined parallel strategy.

8. The method according to any one of claims 1-7, wherein, Also includes: For the target computing unit that interacts with data between multiple data centers within the unit group corresponding to the target parallel computing strategy, the adaptive routing function of the target computing unit is disabled.

9. The method according to claim 8, wherein, Also includes: For the preset collective communication library upon which the communication process between multiple data centers is based, disable the adaptive routing function of the preset collective communication library.

10. The method according to any one of claims 1-9, wherein, The computation and communication processes of the communication data corresponding to the target parallel computing strategy overlap, and The method further includes: The performance degradation rate of the communication data generated by the target parallel computing strategy during long-distance communication between multiple data centers is determined compared to the short-distance communication process within a single data center. Determine the proportion of the duration of the communication process not covered by the computation process in a single iterative training cycle within the communication data generated by the target parallel computing strategy; Based on the performance degradation rate information and the percentage information, the performance loss information of the distributed training process is determined.

11. A distributed training device spanning multiple data centers, comprising: A grouping unit is configured to group computing units in multiple data centers according to the grouping size corresponding to each of the multiple parallel computing strategies, thereby obtaining a unit group corresponding to the parallel computing strategy. The parallel computing strategy is used to split the model to be trained and / or the training sample set for distributed training of the model to be trained. The strategy determination unit is configured to determine a target parallel computing strategy from the multiple parallel computing strategies based on the sensitivity of the communication data generated by each of the multiple parallel computing strategies to the communication latency duration. The communication control unit is configured to, during the distributed training process of the model to be trained, control the unit group corresponding to the target parallel computing strategy to interact the communication data generated by the target parallel computing strategy between multiple data centers, and control the unit group corresponding to other parallel computing strategies to interact the communication data generated by other parallel computing strategies within their respective data centers.

12. The apparatus according to claim 11, wherein, The various parallel computing strategies mentioned include data parallelism, pipelined parallelism, and tensor parallelism. The device further includes: The size-determined unit is configured as follows: Based on the computing architecture of the service device where the computing unit is located, the group size corresponding to the tensor parallel strategy is determined; the group size corresponding to the data parallel strategy and the pipeline parallel strategy are determined by adopting the method that the group size corresponding to the data parallel strategy is greater than the group size corresponding to the pipeline parallel strategy.

13. The apparatus according to claim 12, wherein, The strategy determination unit is further configured to: Since the sensitivity of the data parallel strategy is lower than that of the tensor parallel strategy and the pipeline parallel strategy, the data parallel strategy is determined as the target parallel computing strategy.

14. The apparatus according to any one of claims 11-13, wherein, The communication control unit is further configured to: Multiple first ring communication links are established between multiple unit groups corresponding to the target parallel computing strategy. Each first ring communication link includes a computing unit in each of the multiple unit groups. Some of the training models included in the multiple computing units in the first ring communication link are the same. In the first ring communication link, a global reduction communication method is used to exchange the communication data generated by the target parallel computing strategy.

15. The apparatus according to claim 14, wherein, The communication control unit is further configured to: Based on the transmission rate between computing units in the first ring communication link and the preset redundancy factor, the target transmission rate of the communication channel between the multiple data centers is determined; in the first ring communication link, the communication data generated by the target parallel computing strategy is exchanged using a global reduction communication method and the target transmission rate.

16. The apparatus according to claim 15, wherein, The communication control unit is further configured to: Based on a preset capacity threshold, a target capacity is set for the buffer corresponding to the communication channel between multiple data centers; the communication data generated by the target parallel computing strategy is exchanged using the buffer with the target transmission rate and the target capacity.

17. The apparatus according to claim 13, wherein, The communication control unit is further configured to: A second ring communication link is established based on multiple computing units in the unit group corresponding to the tensor parallel strategy; in the second ring communication link, a global reduction communication method is used to exchange the communication data generated by the tensor parallel strategy. Furthermore, point-to-point communication is used between multiple computing units in the unit group corresponding to the pipelined parallel strategy to exchange the communication data generated by the pipelined parallel strategy.

18. The apparatus according to any one of claims 11-17, wherein, Also includes: The first setting unit is configured to disable the adaptive routing function of the target computing unit that performs data interaction among multiple data centers in the unit group corresponding to the target parallel computing strategy.

19. The apparatus according to claim 18, wherein, Also includes: The second setting unit is configured to disable the adaptive routing function of the preset collective communication library, which is the basis for the communication process between the multiple data centers.

20. The apparatus according to any one of claims 11-19, wherein, The computation and communication processes of the communication data corresponding to the target parallel computing strategy overlap, and The device further includes: The loss determination unit is configured as follows: The system determines the performance degradation ratio of the long-distance communication process between multiple data centers generated by the target parallel computing strategy compared to the short-distance communication process within a single data center; it also determines the proportion of the duration of the communication process not covered by the computing process within an iterative training cycle; and based on the performance degradation ratio and the proportion, it determines the performance loss information of the distributed training process.

21. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product, comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images