Model deployment method, training method, device, and program product

Abstract

The application relates to the technical field of data processing, and discloses a model deployment method, a training method, equipment and a program product. In the method, a model deployment device acquires training information and determines communication time delays between computing units in different data centers. A cross-domain parallel training strategy is determined based on the communication time delay information between the computing units in the different data centers and the training information, and model training is performed based on the cross-domain parallel training strategy. The cross-domain parallel training strategy is determined based on the communication time delay information between the computing units in the different data centers and the training information, and can reduce communication overhead in a model training process in a cross-domain training scenario, thereby improving the efficiency of model training.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a model deployment method, training method, device, and program product. Background Technology

[0002] With the continuous development of deep learning (DL) technology, large models, also known as foundation models, are widely used in fields such as computer vision, natural language processing, autonomous driving, and speech recognition. Large models have a large number of parameters and complex structures. Currently, distributed deep learning (DDL) techniques are commonly used to train large models. For example, within a server cluster / data center composed of graphics processing units (GPUs), multiple GPU servers within a data center can be used to train large models in parallel.

[0003] Due to physical limitations such as available power resources, the number of GPUs available for model training within a single data center is limited. To achieve better results, the number and scale of parameters in large models can be increased, but training a single data center may take a considerable amount of time. Furthermore, in some training scenarios (such as training with financial data), protecting user privacy prevents the transfer of training data distributed across different data centers to a single data center for training. Therefore, to train large models based on training data distributed across different data centers, and to better train large models with a large number of parameters or a large number of parameters, it is necessary to use multiple data centers for parallel training of the model; that is, distributed training across data centers / domains is required.

[0004] However, in distributed training scenarios across data centers / domains, a large amount of intermediate data is generated during model training. The transmission of this sample data and intermediate data between various data centers leads to a large amount of communication overhead, which in turn affects the efficiency of model training. Summary of the Invention

[0005] To address the aforementioned issues, the purpose of this application is to provide a model deployment method, training method, device, and program product.

[0006] The first aspect of this application provides a model deployment method, the method comprising: acquiring training information of a model training system, wherein the model training system is used to train a model to be trained, and the model training system includes multiple data centers, wherein each data center includes at least one computing unit, and the training information includes at least one of topology information of the multiple data centers and data distribution information of training data; determining communication latency information between computing units in different data centers of the multiple data centers; determining a cross-domain parallel training strategy based on the communication latency information and the training information, wherein, under the cross-domain parallel training strategy, computing units in at least two data centers of the model training system perform training of the model to be trained; and sending the cross-domain parallel training strategy to the model training system.

[0007] In this embodiment of the application, the model deployment device determines a cross-domain parallel training strategy based on at least one of the communication latency information between computing units in different data centers in multiple data centers, the topology information of multiple data centers, and the data distribution information of the training data of the model to be trained.

[0008] For example, a model deployment device can determine a cross-domain parallel training strategy based on communication latency information and the topology information of multiple data centers. This determined strategy selects computing units located in different data centers with lower communication latency from the multiple data centers participating in parallel training to execute the computational tasks for the corresponding model training phase. Thus, training the model based on this determined cross-domain parallel training strategy can reduce the communication latency of data transmission between computing units located in different data centers during cross-domain training, thereby reducing model training time and improving model training efficiency.

[0009] For example, a model deployment device can determine a cross-domain parallel training strategy based on communication latency information and the data distribution information of the training data. The determined strategy can then distribute the training data to computing units in data centers corresponding to the training data distribution or in data centers with low communication latency between them. Thus, training the model based on this strategy reduces the communication time required to send training data from data centers corresponding to the training data distribution to data centers where the training data is not distributed, as well as the communication time during gradient aggregation and synchronization, thereby reducing model training time and improving training efficiency.

[0010] It is understandable that a determined cross-domain parallel training strategy can select computing units located in different data centers with lower communication latency from multiple data centers participating in parallel training to perform the computational tasks of the corresponding model training phase. Furthermore, the determined cross-domain parallel training strategy can also distribute training data to computing units in data centers with corresponding distributions of training data or data centers with lower communication latency between them.

[0011] Thus, by training the model based on the determined cross-domain parallel training strategy, the communication latency of data transmission between computing units located in different data centers during cross-domain training can be reduced. Furthermore, the communication time of sending training data from the data center corresponding to the distribution of training data to the data center where the training data is not distributed, as well as the communication time during gradient aggregation and synchronization, can be reduced, thereby reducing model training time and improving model training efficiency.

[0012] In one possible implementation of the first aspect described above, determining the communication latency information between computing units in different data centers within a plurality of data centers includes: measuring the communication latency between computing units in different data centers based on a network performance testing tool; or, obtaining the communication latency between computing units in different data centers based on a communication operator, wherein the communication operator includes a field for measuring the communication latency between computing units.

[0013] It is understandable that obtaining communication latency between computing units in different data centers based on communication operators involves measuring the communication latency between computing units during data transmission using the communication operators. Compared to methods that measure communication latency between computing units in different data centers using network performance testing tools, the measurement results obtained by measuring the communication latency between computing units using communication operators are more accurate.

[0014] In one possible implementation of the first aspect described above, the communication latency between computing units in different data centers includes: a first communication latency between computing units located in different data centers across multiple data centers.

[0015] In this embodiment of the application, the communication latency between computing units in different data centers can be determined by the following method: based on the communication operator, the first communication latency between each computing unit located in different data centers in multiple data centers is measured to obtain a first communication latency matrix, and each first communication latency in the first communication latency matrix is ​​used as the communication latency between computing units in different data centers.

[0016] It is understandable that by iterating through the communication latency between each computing unit in each data center, the specific communication latency between each computing unit in different data centers can be accurately obtained.

[0017] In one possible implementation of the first aspect described above, the communication latency between computing units in different data centers includes: the average of the second communication latency between computing units in the respective computing unit samples located in different data centers across multiple data centers.

[0018] In this embodiment of the application, the communication latency between computing units in different data centers can be determined by the following method: selecting multiple computing units from each data center as computing unit samples of each data center, measuring the second communication latency between each computing unit in the different computing unit samples based on the communication operator, and taking the average value of the second communication latency between each computing unit in the different computing unit samples as the communication latency between computing units in different data centers.

[0019] It is understandable that the number of computational units is usually large in cross-domain training scenarios. By sampling (determining a sample of computational units) and determining the communication latency between computational units in each sample, the measurement time and cost of communication latency can be reduced.

[0020] In one possible implementation of the first aspect described above, the communication latency between computing units in different data centers includes: the average of the third communication latency between computing units in various type groups in different data centers across multiple data centers, wherein the type groups are based on the type of computing units in each data center.

[0021] In this embodiment of the application, the communication latency between computing units in different data centers can be determined by the following method: the computing units in multiple data centers are divided into multiple type groups based on the type of computing unit; the third communication latency between computing units in each type group of different data centers is obtained based on the communication operator; and the average value of the third communication latency between multiple computing units in each type group of different data centers is taken as the communication latency between computing units in each type group of different data centers.

[0022] It is understandable that data centers may contain different types of data units. The performance of different types of data units may vary. By dividing the computing units in multiple data centers into multiple types of groups and measuring the communication latency of the computing units in each type of group, the communication latency between the corresponding different types of data units can be obtained.

[0023] In one possible implementation of the first aspect described above, each data center includes multiple computing units, each computing unit corresponding to at least one computing node; and the communication latency between computing units in different data centers includes the average of the third communication latency between each computing unit in each computing node of each of the multiple data centers.

[0024] In this application embodiment, the communication latency between computing units in different data centers can be determined by the following method: measuring the fourth communication latency between computing units in each computing node located in different data centers based on the communication operator; and taking the average of the fourth communication latency between computing units in each computing node in different data centers as the communication latency between computing units in each computing node in different data centers.

[0025] It is understandable that each data center may contain multiple computing nodes. The distance between different computing nodes may vary. By measuring the communication latency between computing units within each computing node, the communication latency between data units corresponding to different computing nodes can be obtained.

[0026] In one possible implementation of the first aspect described above, the cross-domain parallel training strategy includes at least one of the following: a cross-domain pipelined parallel training strategy, wherein at least two different pipeline stages of the model to be trained are assigned to computing units in different data centers, wherein one pipeline stage includes at least one network layer of the model to be trained; and a cross-domain data parallel training strategy, wherein at least two sub-training data are assigned to computing units in different data centers, wherein the sub-training data is a subset of the training data of the model to be trained, and there is no overlap between the sub-training data.

[0027] In one possible implementation of the first aspect described above, the cross-domain parallel training strategy also includes parallel training strategies within each data center corresponding to the cross-domain parallel training strategy, wherein the parallel training strategies within each data center include at least one of the following: tensor parallel training strategy, sequence parallel training strategy, and expert parallel training strategy.

[0028] In one possible implementation of the first aspect described above, determining a cross-domain parallel training strategy based on communication latency information and training information includes: determining at least one candidate orchestration domain based on communication latency information and / or training information, wherein the candidate orchestration domain includes a candidate pipeline parallel orchestration domain and / or a candidate data parallel orchestration domain, and the candidate orchestration domain includes at least two data centers and / or multiple computing units in at least two data centers; and determining a cross-domain parallel training strategy based on communication latency information, training information, and at least one candidate orchestration domain.

[0029] In one possible implementation of the first aspect described above, the communication latency between computing units located in different data centers within the candidate pipeline parallel orchestration domain is less than a communication latency threshold.

[0030] In this embodiment of the application, candidate pipeline parallel orchestration domains can be determined by the following method: sorting the communication latency between computing units in every two data centers in multiple data centers in ascending order, and assigning the computing units corresponding to the first K communication latencies to the same candidate pipeline parallel orchestration domain, where K is an integer greater than 1; or, assigning the computing units corresponding to communication latencies less than the communication latency threshold in every two data centers in multiple data centers to the same candidate pipeline parallel orchestration domain.

[0031] It is understandable that by controlling the communication latency between computing units located in different data centers within the candidate pipelined parallel orchestration domain to be less than a communication latency threshold, the communication latency between computing units in different data centers will be minimized when determining the cross-domain pipelined parallel training strategy based on training information, communication latency information, and the candidate pipelined parallel orchestration domain. When training the model based on the determined cross-domain pipelined parallel training strategy, the communication latency for data transmission between computing units located in different data centers during cross-domain training can be reduced, thereby reducing model training time and improving model training efficiency.

[0032] In one possible implementation of the first aspect described above, the candidate data parallel orchestration domain corresponds to different data centers that contain training data from the same users.

[0033] It is understandable that by controlling the distribution of training data from the same users across different data centers within the candidate data parallel orchestration domain, when determining the cross-domain data parallel training strategy based on training information, communication latency information, and the candidate data parallel orchestration domain, the corresponding data center can use the training data distributed within that data center. When training the model based on the determined cross-domain data parallel training strategy, the time spent sending training data to other data centers with undistributed training data can be reduced during cross-domain training, thereby reducing model training time and improving model training efficiency.

[0034] In this embodiment of the application, the candidate data parallel orchestration domain can be determined by the following method: dividing the data centers corresponding to the training data of the same user into the same candidate data parallel orchestration domain.

[0035] In one possible implementation of the first aspect described above, the cross-domain pipelined parallel training strategy includes at least one of the following parameters: the pipeline stage of the model to be trained executed for each data center, the pipeline partitioning method of the model to be trained, the size of the data processed by each computing unit in each data center, and the scheduling method of the communication operators between the data centers; or, the cross-domain data parallel training strategy includes at least one of the following parameters: the sub-training data for each data center, the partitioning strategy of the training data, the communication algorithm for gradient aggregation and synchronization between the data centers, and the parameter configuration information of the communication algorithm for gradient aggregation and synchronization between the data centers.

[0036] In this embodiment, the S / R communication operator's stream scheduling scheme is configured as dual-stream scheduling, which enables synchronous data transmission and reception, thereby improving the training efficiency of the model based on the pipelined parallel training strategy. By evaluating various communication algorithms and their configuration information, a more efficient communication algorithm and its corresponding parameter configuration information, such as timing and grouping configuration information, are determined, which further improves the training efficiency of the model based on the data parallel training strategy.

[0037] In one possible implementation of the first aspect described above, the sum of the communication times between the various computing units corresponding to the cross-domain parallel training strategy is less than a first threshold.

[0038] In one possible implementation of the first aspect described above, the topology information of the multiple data centers includes at least one of the following: the distance between the multiple data centers, the topology between the multiple data centers, and the topology between multiple computing units within each of the multiple data centers; or, the data distribution information of the training data includes at least one of the following: the data centers corresponding to the distribution of the training data, and the amount of training data corresponding to the distribution of each data center.

[0039] In one possible implementation of the first aspect described above, the training information further includes: model information of the model to be trained, wherein the model information of the model to be trained includes at least one of the following: the number of layers of the model to be trained, and the model parameters of the model to be trained.

[0040] A second aspect of this application provides a model training method applied to a model training system comprising multiple data centers, wherein each data center includes at least one computing unit; and the method comprises: obtaining a cross-domain parallel training strategy from a model deployment device, wherein the cross-domain parallel training strategy is determined by the model deployment device based on communication latency information between computing units in different data centers and training information of the model training system, wherein the training information includes at least one of topology information of the multiple data centers and data distribution information of the training data; and under the cross-domain parallel training strategy, computing units in at least two data centers in the model training system perform training of the model to be trained; and performing multiple rounds of iterative training on the model to be trained based on the cross-domain parallel training strategy.

[0041] In one possible implementation of the second aspect described above, the method further includes: during the training process, determining the sum of communication delays between the various computing units corresponding to the cross-domain parallel training strategy; and when the sum of communication delays between the various computing units corresponding to the cross-domain parallel training strategy is greater than or equal to a first threshold, adjusting the cross-domain parallel training strategy so that the sum of communication delays between the various computing units corresponding to the adjusted cross-domain parallel training strategy is less than the first threshold.

[0042] A third aspect of this application provides a model deployment device, comprising: an information acquisition module for acquiring training information of a model training system, wherein the model training system is used to train a model to be trained, and the model training system includes multiple data centers, wherein each data center includes at least one computing unit, and the training information includes at least one of topology information of the multiple data centers and data distribution information of the training data; the information acquisition module is further configured to determine communication latency information between computing units in different data centers of the multiple data centers; a parallel orchestration module for determining a cross-domain parallel training strategy based on the communication latency information and the training information, wherein, under the cross-domain parallel training strategy, computing units in at least two data centers of the model training system execute the training of the model to be trained; and a training module for sending the cross-domain parallel training strategy to the model training system.

[0043] A fourth aspect of this application provides a model deployment apparatus, comprising: a memory for storing instructions executed by one or more processing units of the model deployment apparatus; and a processing unit, one of the processing units of the model deployment apparatus, for executing the instructions stored in the memory to implement any of the methods of the first aspect described above.

[0044] The fifth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a model deployment device, cause the model deployment device to implement any of the methods described in the first aspect above.

[0045] The sixth aspect of this application provides a computer program product including instructions that, when executed on a model deployment device, cause the model deployment device to implement any of the methods described in the first aspect above.

[0046] The seventh aspect of this application provides a model training system comprising: multiple data centers, wherein one data center includes at least one computing node. Attached Figure Description

[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1A An embodiment of this application illustrates a schematic diagram of communication time for pipelined parallel training within a single data center;

[0049] Figure 1B An embodiment of this application illustrates a schematic diagram of communication time for pipelined parallel training across a data center;

[0050] Figure 2 An embodiment of this application illustrates a performance comparison diagram of different communication algorithms;

[0051] Figure 3 A schematic diagram of a cross-domain training scenario is shown according to an embodiment of this application;

[0052] Figure 4 A schematic flowchart of a model deployment method is shown according to an embodiment of this application;

[0053] Figure 5 A flowchart illustrating a communication delay measurement method is shown according to an embodiment of this application;

[0054] Figure 6 A flowchart illustrating another communication delay measurement method is shown according to an embodiment of this application;

[0055] Figure 7 A flowchart illustrating a candidate domain orchestration method is shown according to an embodiment of this application;

[0056] Figure 8 A schematic diagram of the structure of a model deployment device 100 is shown according to an embodiment of this application;

[0057] Figure 9A schematic flowchart of a model training method is shown according to an embodiment of this application;

[0058] Figure 10 A schematic diagram of the structure of a model training system 200 is shown according to an embodiment of this application;

[0059] Figure 11 A schematic diagram of the hardware structure of a model deployment device 100 is shown according to an embodiment of this application;

[0060] Figure 12 A logical block diagram of a model deployment and model training process is shown according to an embodiment of this application;

[0061] Figure 13 An embodiment of this application illustrates a logic block diagram for automatic parallel orchestration. Detailed Implementation

[0062] The illustrative embodiments of this application include, but are not limited to, a model deployment method, a training method, an apparatus, and a program product.

[0063] Before introducing the technical solutions involved in the embodiments of this application, some of the terms included in the embodiments of this application will be explained.

[0064] (1) Pipeline parallel technology

[0065] Pipeline parallelism (PP) is a parallel training strategy applied to artificial intelligence model training, and it is widely used in distributed model training. Pipeline parallelism distributes different parts of the model (such as layers of a neural network) across different computing devices (or computing units), allowing multiple devices to process the model's training data layer by layer in a pipeline manner.

[0066] For example, taking the backpropagation (BP) neural network model as an example, the training process of the BP neural network model is mainly divided into two stages: forward propagation and backward propagation. In the forward propagation stage, the training sample data is input into the BP neural network model, and the predicted output is obtained through the calculation of the hidden layers and the output layer. The error is calculated by comparing it with the actual target output. In the backward propagation stage, based on the error, the error is propagated layer by layer from the output layer to the hidden layers through the gradient descent algorithm, and the weights and biases are continuously adjusted to reduce the error.

[0067] For training a backpropagation (BP) neural network model, different layers of the BP neural network model can be assigned to different computing units for pipelined parallel training. These computing units can be accelerators such as graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), and digital signal processing units (DSPs). Computing units can also be servers such as GPU servers, NPU servers, and TPU servers; a single server can include multiple accelerators.

[0068] For example, Figure 1A A schematic diagram of the communication time for pipelined parallel training within a single data center is shown.

[0069] like Figure 1A As shown, suppose the four layers of a BP neural network model A (hereinafter referred to as Model A) are distributed to four GPUs in the same data center for training. For example, the input layer is distributed to GPU1, the hidden layer to GPU2, the output layer to GPU3, and the feedback layer to GPU4. During the input training data process, the training data of Model A can be divided into eight equal-sized data blocks, i.e., eight micro-batches: data block 1, data block 2, data block 3, data block 4, data block 5, data block 6, data block 7, and data block 8. Each data block is then sequentially input into GPU1 as the input data for the input layer. GPU1 runs the first stage of the pipeline (stage 1), GPU2 runs the second stage of the pipeline (stage 2), GPU3 runs the third stage of the pipeline (stage 3), and GPU4 runs the fourth stage of the pipeline (stage 4).

[0070] Because of the dependencies between layers in Model A, each pipeline stage needs to wait for the training output data from the previous pipeline stage to be transferred to the current stage. For example, in the forward propagation stage, GPU1 transfers data block 1 to GPU2. While GPU1 is sending data block 1 to GPU2, GPU2 needs to wait, meaning it doesn't perform any work. At this time, the pipeline for the second stage corresponding to GPU2 will pause execution for one clock cycle, resulting in a bubble in the task queue of the second stage pipeline. A bubble in a pipeline stage is an idle operation (an operation that does not perform any processing) caused by a pipeline stage pause. One bubble is typically used to indicate an idle operation within one clock cycle. That is, the pipeline stage containing the bubble is idle during the clock cycle corresponding to the bubble. While GPU2 is waiting for GPU1 to send data block 1 to GPU2, GPU3 also needs to wait. At this time, the pipeline for the third stage corresponding to GPU3 will pause execution for one clock cycle, resulting in a bubble in the task queue of the third stage pipeline. While GPU2 is sending data block 1 to GPU3, GPU3 still needs to wait. At this time, the third stage pipeline corresponding to GPU3 will continue to pause execution for 1 clock cycle, resulting in another vacancy in the task queue of the third stage pipeline.

[0071] And so on, Figure 1A The total communication time (total_time) for training model A using the four GPUs in a pipelined parallel manner is 33 clock cycles. Stage 1, stage 2, stage 3, and stage 4 each generate 9 bubble times (bubble_time), meaning that the idle time is 9 clock cycles.

[0072] For example, Figure 1B A schematic diagram of the communication time for pipelined parallel training across a data center is shown.

[0073] like Figure 1B As shown, assume that the four layers of the BP neural network model A are distributed to GPU1 and GPU2 in data center (DC) 1, and GPU3 and GPU4 in DC2. The training data for model A is still divided into eight equal-sized data blocks, i.e., eight micro-batches: data block 1, data block 2, data block 3, data block 4, data block 5, data block 6, data block 7, and data block 8. GPU1 runs pipeline stage 1, GPU2 runs pipeline stage 2, GPU3 runs pipeline stage 3, and GPU4 runs pipeline stage 4.

[0074] Because of the dependencies between layers in Model A, each pipeline stage needs to wait for the training output data from the previous pipeline stage to be transferred to the current stage. For example, in the forward propagation stage, GPU1 transfers data block 1 to GPU2. While GPU1 is sending data block 1 to GPU2, GPU2 needs to wait. At this time, the pipeline for the second stage corresponding to GPU2 will pause execution for one clock cycle, resulting in one empty bubble in the task queue of the second stage pipeline. While GPU2 is waiting for GPU1 to send data block 1 to GPU2, GPU3 also needs to wait. At this time, the pipeline for the third stage corresponding to GPU3 will pause execution for one clock cycle, resulting in one empty bubble in the task queue of the third stage pipeline. While GPU2 is sending data block 1 to GPU3, GPU3 still needs to wait. At this time, the pipeline for the third stage corresponding to GPU3 will continue to pause execution for one clock cycle, resulting in another empty bubble in the task queue of the third stage pipeline.

[0075] Furthermore, because GPU2 and GPU3 are located in different data centers, the communication time for data transmission and reception between GPU2 and GPU3 is relatively long, for example... Figure 1B The communication exposure time shown is (comm_exposure_time).

[0076] It is understandable that GPU3 incurs a one-clock-cycle forward communication exposure time during the process of receiving data block 1 from GPU2, i.e., the communication exposure time during the forward propagation process. GPU4 still needs to wait for the corresponding clock cycle of this forward communication exposure time. Therefore, the fourth stage pipeline corresponding to GPU4 will pause execution for one clock cycle, resulting in one empty bubble in the task queue of the fourth stage pipeline. Thus, before GPU4 receives data block 1, the fourth stage pipeline corresponding to GPU4 needs to pause execution for four clock cycles, resulting in four empty bubbles in the task queue of the fourth stage pipeline.

[0077] However, as mentioned above Figure 1A As shown, during pipelined parallel training within a single data center, before GPU4 receives data block 1, the pipeline for the fourth stage corresponding to GPU4 only needs to pause execution for 3 clock cycles, and the task queue of the fourth stage pipeline only generates 3 empty bubbles. In other words, due to the communication exposure time between data units across data centers (e.g., GPU2 and GPU3), compared to pipelined parallel training within a single data center, pipelined parallel training across data centers not only increases communication exposure time but also increases empty bubble time.

[0078] And so on, Figure 1BThe four GPUs shown require 41 clock cycles to train model A using a pipelined parallel approach. Stage 1 generates 17 cavitation bubbles, stage 2 generates 9 cavitation bubbles, stage 3 generates 9 cavitation bubbles, and stage 4 generates 17 cavitation bubbles. The communication exposure time for stages 1 and 4 is 0, while the communication exposure time for stages 2 and 3 is 8 (the reverse communication exposure time for stage 2 is 8, and the forward communication exposure time for stage 3 is 8). That is, the data communication between stages 2 and 3 corresponds to the data communication between GPU2 and GPU3 across data centers, resulting in increased round-trip communication latency.

[0079] Understandable, compared to Figure 1A The pipelined parallel training within a single data center is shown. Figure 1B The pipelined parallel training process shown in the data center example adds 16 clock cycles of communication exposure time and 16 clock cycles of idling time.

[0080] (2) Data Parallelism Technology

[0081] Data parallelism (DP) is a parallel training strategy applied to artificial intelligence model training, and it is widely used in distributed model training. Data parallelism deploys the same model on multiple different computing devices (or computing nodes), divides the model's training data into different batches, and has each computing device process the training data of each batch in parallel. Then, at the end of each training iteration, gradient aggregation and synchronization are performed among the multiple computing devices, and the model parameters are updated.

[0082] Multiple computing devices can perform gradient aggregation and synchronization through the allreduce (AR) communication primitive. The allreduce communication primitive can be implemented using various communication algorithms, such as ring algorithms, nonuniform hierarchical ring (NHR) algorithms, and large number multiplication algorithms (e.g., half doubling (HD) algorithms).

[0083] It is understandable that different communication algorithms may have different performance characteristics. For example, for the same model, different communication algorithms have different computational rules, and the time required to complete gradient aggregation and synchronization may differ, as may the computational resources used. Therefore, the training efficiency of models using different communication algorithms may also differ. For example, taking the total communication time for gradient aggregation and synchronization and the amount of cross-domain data (cross-domain traffic) as a standard for measuring the performance of a common algorithm, assuming that 8 GPUs are divided into 2 domains (2 data centers), each domain includes 4 GPUs, and data-parallel distributed model training is performed, the communication data volume of a single GPU is S, the communication bandwidth is B, the convergence ratio is 64, the bandwidth of a single GPU is B / 4, and the cross-domain bandwidth is B / 64; then, the communication time and cross-domain traffic for gradient aggregation and synchronization based on different communication algorithms can be referenced... Figure 2 The diagram shows a performance comparison of different communication algorithms.

[0084] like Figure 2 As shown, if gradient aggregation and synchronization are performed based on the ring algorithm, the algorithm includes:

[0085] The communication time for synchronizing the residual sum of squares (rs) in Step 1-7 is: S / 8 / (B / 64)*7=56S / B.

[0086] The communication time for synchronizing the average gradient (ag) in Step 8-14 is: S / 8 / (B / 64)*7=56S / B.

[0087] Therefore, the total communication time for gradient aggregation and synchronization is 112 seconds / second, and the cross-domain traffic is 8 seconds / 14 seconds.

[0088] Or, for example, if gradient aggregation and synchronization are based on the NHR algorithm, the algorithm includes:

[0089] The communication time for synchronizing RS in Step 1-3 is: S / 4*4 / B*3=3S / B.

[0090] The communication time for Step 4 cross-domain autoregressive (ar) is: S / 4*4 / (B / 16)=64S / B.

[0091] The communication time for synchronizing ag in Step 5-7 is: S / 4*4 / B*3=3S / B.

[0092] Therefore, the total communication time for gradient aggregation and synchronization is 70 seconds per second, and the cross-domain traffic is 4 seconds per second.

[0093] Or, for example, if gradient aggregation and synchronization are performed based on the far-to-near HD algorithm, the algorithm would include:

[0094] The communication time for cross-domain synchronization of rs in Step 1 is: S / 2*4 / (B / 64)=128S / B.

[0095] The communication time for Step 2 to synchronize rs is: S / 4*4 / B=S / B.

[0096] The communication time for synchronizing RS in Step 3 is: S / 8*4 / B=S / 2B.

[0097] The communication time for Step 4 to synchronize ag is: S / 8*4 / B=S / 2B.

[0098] The communication time for Step 5 to synchronize ag is: S / 4*4 / B=S / B.

[0099] The communication time for Step 6 cross-domain synchronization of ag is: S / 2*4 / (B / 64)=128S / B.

[0100] Therefore, the total communication time for gradient aggregation and synchronization is 259 seconds, and the cross-domain traffic is 8 seconds.

[0101] Or, for example, if gradient aggregation and synchronization are performed based on the near-to-far HD algorithm, the algorithm includes:

[0102] The communication time for synchronizing RS in Step 1 is: S / 2*4 / B=2S / B.

[0103] The communication time for Step 2 to synchronize rs is: S / 4*4 / B=S / B.

[0104] The communication time for cross-domain synchronization of rs in Step 3 is: S / 8*4 / (B / 64)=32S / B.

[0105] The communication time for Step 4 cross-domain synchronization of ag is: S / 8*4 / (B / 64)=32S / B.

[0106] The communication time for Step 5 to synchronize ag is: S / 4*4 / B=S / B.

[0107] The communication time for Step 6 to synchronize ag is: S / 2*4 / B=2S / B.

[0108] Therefore, the total communication time for gradient aggregation and synchronization is 70 seconds per second, and the cross-domain traffic is 8 seconds per second.

[0109] It is understandable that the communication time between computing units within the same data center differs from that between computing units in different data centers. Therefore, using communication algorithms with different computational rules to perform gradient aggregation and synchronization on the same model may require different communication times and different amounts of communication data.

[0110] However, based on the above Figure 1B As can be seen from the pipeline execution process and the algorithm execution process described above, both the pipeline parallelism and data parallelism techniques suffer from the drawback of increased communication latency during training due to the increased communication latency between different data centers. Therefore, this application aims to solve this problem. To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described clearly and in detail below with reference to the accompanying drawings.

[0111] The model deployment method provided in this application can be applied to cross-data center / cross-domain distributed training (hereinafter referred to as cross-domain training scenario), which may include multiple data centers (DCs). For example, Figure 3 A schematic diagram of a cross-domain training scenario is shown according to an embodiment of this application.

[0112] like Figure 3 As shown, DC1 includes multiple computing units, and DC2 includes multiple computing units. These computing units can be servers of various types, such as GPU servers, NPU servers, and TPU servers.

[0113] It is understandable that computing units can also be accelerator cards (or data processing units) of the type GPU, NPU, TPU, DSP, etc. When the computing unit is an accelerator card, multiple computing units can be located in the same computing node in the same data center, where the computing node can be a server of the type GPU server, NPU server, TPU server, etc.

[0114] For ease of description, the following uses the example of a server (e.g., a GPU server as the computing unit and a GPU as the computing node) to illustrate the technical solutions in the embodiments of this application.

[0115] Different data centers are typically configured in different geographical regions. For example, such as Figure 3 As shown, DC1 and DC2 are configured in two regions hundreds or thousands of kilometers apart. Correspondingly, the data transmission distance between them is also relatively long, resulting in significant data transmission latency. Given the high latency inherent in the aforementioned pipelined parallelism and data parallelism techniques...

[0116] As mentioned earlier, existing parallel training strategies (such as pipelined parallelism and data parallelism) are mainly applied in single-datacenter training scenarios. Therefore, they do not consider the impact of increased communication time due to distance and bandwidth differences between different data centers on model training efficiency in cross-domain training scenarios. In other words, in cross-domain training scenarios, each layer generates a large amount of intermediate data during model training (such as the aforementioned...). Figure 1B The data blocks transmitted between the various pipelines (as shown) generate significant communication overhead during the transmission of these intermediate data between data centers, thus affecting the efficiency of model training.

[0117] The reasons for this can be found in the above. Figure 1B Because GPU2 and GPU3 are located in different data centers, communication exposure time occurs during data transmission and reception between them. Furthermore, communication exposure time in one pipeline stage may increase the bubbling time of the next pipeline stage. For example, if GPU3 experiences a one-clock-cycle communication exposure time while receiving data block 1 from GPU2, GPU4 still needs to wait for the corresponding clock cycle. Therefore, the fourth stage pipeline corresponding to GPU4 will pause execution for one clock cycle, resulting in a bubbling in the task queue of the fourth stage pipeline. If a large amount of communication exposure time occurs during model training, it will increase the communication overhead during model training, thus affecting the efficiency of model training. (See the above...) Figure 2 The communication time between computing units within the same data center differs from that between computing units in different data centers. Increased communication time between data centers increases communication overhead during model training, thus affecting training efficiency.

[0118] In view of this, embodiments of this application provide a model deployment method applied to cross-domain training scenarios. In this method, the model deployment device acquires training information from the model training system (e.g., topology information of multiple data centers of the model training system, data distribution information of training data of the model to be trained, etc.), and determines the communication latency between computing units in different data centers (e.g., determining...). Figure 3 The communication latency between each GPU in each GPU server in DC1 and each GPU in each GPU server in DC2 is shown. Based on the determined communication latency information and training information, a cross-domain parallel training strategy is determined. Under the cross-domain parallel training strategy, computing units in at least two data centers in the model training system execute the training of the model to be trained.

[0119] For example, a model deployment device can determine a cross-domain parallel training strategy based on communication latency information and the topology information of multiple data centers. This strategy can then select computing units with lower communication latency from the multiple data centers participating in parallel training to perform the computational tasks of the corresponding model training phase. In this way, the communication latency of data transmission between computing units located in different data centers can be reduced during cross-domain training, thereby reducing model training time and improving model training efficiency.

[0120] For example, a model deployment device can determine a cross-domain parallel training strategy based on communication latency information and the data distribution information of the training data. This strategy can then allocate training data to computing units in data centers corresponding to the training data distribution or in data centers with low communication latency between them. This reduces the communication time required to send training data from data centers corresponding to the training data distribution to data centers where the training data is not distributed, as well as the communication time during gradient aggregation and synchronization, thereby reducing model training time and improving model training efficiency.

[0121] For example, the model deployment device determines a cross-domain parallel training strategy based on communication latency information, the topology information of multiple data centers, and the data distribution information of the training data. This strategy can either select computing units located in different data centers with lower communication latency from the multiple data centers participating in parallel training to perform the corresponding computational tasks in the model training phase; or it can distribute the training data to computing units in data centers corresponding to the training data distribution or data centers with lower communication latency between them. Thus, model training based on the determined cross-domain parallel training strategy can reduce the communication latency of data transmission between computing units located in different data centers during cross-domain training. Furthermore, it can reduce the communication time of sending training data from data centers corresponding to the training data distribution to data centers where the training data is not distributed, as well as the communication time during gradient aggregation and synchronization processes, thereby reducing model training time and improving model training efficiency.

[0122] Specifically, cross-domain parallel training strategies include: cross-domain pipeline parallel training strategies between multiple data centers and / or cross-domain data parallel training strategies between multiple data centers.

[0123] For example, based on the communication latency between computing units in different data centers, K groups of computing units with the lowest communication latency are selected from multiple data centers to run different pipeline stages of the model to be run; that is, the model is pipelined in parallel. In this way, the communication latency for transmitting data between pipeline stages of computing units located in different data centers can be reduced during pipelined parallel training.

[0124] For example, based on the distribution information of the training data and the distance between different data centers, the training data can be allocated to computing units in data centers corresponding to the data data's distribution for training, or to computing units in data centers that are closer to the data centers corresponding to the data data's distribution for training. This can reduce the communication latency of data transmission during gradient aggregation and synchronization between computing units located in different data centers during parallel training.

[0125] Understandably, other data can be used to determine the communication algorithm used in the data-parallel training process, the pipeline splitting method in the pipeline parallel training process, etc., to further reduce the communication time when running the model to be trained based on this strategy.

[0126] It is understandable that the cross-domain parallel training strategy is determined based on the communication latency information and training information between computing units in different data centers, and the communication time of the cross-domain parallel training strategy is less than a first threshold. That is, it can reduce the communication overhead of the model training process in cross-domain training scenarios, thereby improving the efficiency of model training.

[0127] In some embodiments, the training information of the model training system includes, but is not limited to: topology information of multiple data centers, data distribution information of training data, model information of the model to be trained, and training parameters configured by the user.

[0128] The topology information of multiple data centers includes, but is not limited to: the distance between multiple data centers, the topology between multiple data centers, the topology between multiple computing units within each data center, the network convergence ratio between multiple data centers, the network bandwidth between multiple data centers, and the fixed latency between multiple data centers.

[0129] The data distribution information of the training data includes, but is not limited to: the geographical distribution information of the training data used for model training (i.e., the data centers corresponding to the distribution of the training data), the amount of training data corresponding to each region (i.e., the amount of training data corresponding to each data center), and other data information.

[0130] The model information of the model to be trained includes, but is not limited to: the number of model layers, model parameters, hidden layer size (the number of neurons in the hidden layer), etc.

[0131] User-configured training parameters include, but are not limited to: the location of the data center specified by the user, and user-set training hyperparameters (such as training accuracy, parallel training strategy, data access permissions, etc.).

[0132] In some embodiments, the cross-domain parallel training strategy can be determined using the following methods:

[0133] Based on communication latency information and training data distribution information, candidate orchestration domains are determined. These candidate orchestration domains include candidate data parallel orchestration domains and candidate pipelined parallel orchestration domains, each containing information about at least two data centers and / or computing units within those data centers. Then, cross-domain data parallel orchestration or cross-domain pipelined parallel orchestration is performed on each candidate orchestration domain to obtain the orchestration results, i.e., cross-domain pipelined parallel training strategies and / or cross-domain data parallel training strategies across multiple data centers.

[0134] Among them, the cross-domain pipeline parallel training strategy between multiple data centers includes, but is not limited to, parameters such as: micro batch size (MBS), virtual pipeline parallel pipeline (vPP), cross-domain inter-layer partitioning method, cross-domain load balancing strategy, and scheduling method of communication operators.

[0135] It is understandable that, considering the different communication algorithms used for gradient aggregation and synchronization in data-parallel training strategies, the communication time and amount of communication data may vary, for example, as mentioned above. Figure 2 The performance comparison of different communication algorithms for gradient aggregation and synchronization of the same model shown may lead to different efficiencies in training the same model using data parallel training strategies with different communication algorithms.

[0136] Therefore, the cross-domain data parallel training strategy can also select and configure various communication algorithms and their configuration information to determine the most efficient communication algorithms and their corresponding parameter configuration information, such as timing and grouping configuration information.

[0137] Cross-domain data parallel training strategies between multiple data centers include, but are not limited to: training data splitting strategies, communication algorithms for gradient aggregation and synchronization between data centers, and parameter configuration information (such as timing, grouping, etc.) of the communication algorithms for gradient aggregation and synchronization between data centers.

[0138] In some embodiments, the communication latency between computing units in different data centers can be measured using network performance testing tools (such as iperf, IFIT, etc.).

[0139] In other embodiments, relevant fields can be added to the communication operators corresponding to the parallel training strategy (e.g., pipelined parallel send / receive (S / R) communication operators, data parallel AR communication operators), and the communication latency between computing units in different data centers can be measured based on the communication operators.

[0140] To better understand the technical solutions of the embodiments of this application, the following will be used as examples. Figure 3 Taking the scenario shown as an example, we will introduce some of the technical solutions of this application in detail.

[0141] Figure 4 A flowchart illustrating a model deployment method is shown according to an embodiment of this application. It can be understood that... Figure 4 The process shown is executed by the model deployment device 100. For simplicity, the following description... Figure 4 The execution entity will not be described again when the process is shown.

[0142] like Figure 4 As shown, this process includes, but is not limited to:

[0143] S401: Obtain training information from model training system 200.

[0144] In some embodiments, the model deployment device 100 obtains training information from the model training system 200 through an information interface. For example, it obtains model information of the model to be trained through a model information interface, and obtains topology information of multiple data centers of the model training system 200 through a topology and physical environment information interface, such as topology information (e.g., the topology between multiple data centers, the topology between multiple computing units within each data center, etc.) and physical environment information (e.g., the network convergence ratio between multiple data centers, the network bandwidth between multiple data centers, and the fixed latency between multiple data centers).

[0145] It is understood that the model training system 200 includes multiple data centers, where each data center includes at least one computing unit (e.g., a GPU).

[0146] In some embodiments, training information includes, but is not limited to, topology information of multiple data centers of the model training system 200, user-configured training parameters, data distribution information of training data, and model information of the model to be trained.

[0147] The topology information of multiple data centers includes, but is not limited to: the distance between multiple data centers, the topology between multiple data centers, the topology between multiple computing units within each data center, the network convergence ratio between multiple data centers, the bandwidth between computing units within each data center and the bandwidth between computing units between different data centers (referred to as intra-domain and inter-domain bandwidth), and the physical parameters (such as bandwidth and fixed latency) of forwarding devices between multiple data centers. Forwarding devices include, but are not limited to, switches, routers, and optical transport networks (OTNs).

[0148] User-configured training parameters include, but are not limited to: the location of the data center specified by the user (e.g., for a user with a data center located in city A, specifying that the model is trained in a data center located in city A and a data center located in city B), and user-set training hyperparameters (e.g., training accuracy, parallel training strategy, data access permissions, etc.).

[0149] The data distribution information of the training data includes, but is not limited to: the geographical distribution information of the training data used for model training (e.g., information on multiple data centers corresponding to the training data), the amount of training data corresponding to each region (e.g., the amount of training data distributed in each data center), and other data information.

[0150] The model information of the model to be trained includes, but is not limited to: the number of model layers, model parameters, and hidden layer size (e.g., the number of neurons in the hidden layer).

[0151] S402: Determine the communication latency information between computing units in different data centers of the multiple data centers of the model training system 200.

[0152] In some embodiments, the model deployment device 100 measures the communication latency between computing units in different data centers within the model training system 200 based on network performance testing tools (such as iperf, IFIT, etc.).

[0153] In other embodiments, the model deployment device 100 adds relevant fields to the communication operators corresponding to the parallel training strategy (e.g., pipelined parallel S / R communication operators, data parallel AR communication operators), and measures the communication latency between computing units in different data centers in the model training system 200 based on the communication operators.

[0154] For example, the communication latency between computing units in different data centers includes, but is not limited to: the first communication latency between computing units in different data centers in multiple data centers of the model training system 200; the average of the second communication latency between computing units in computing unit samples in different data centers in multiple data centers of the model training system 200; the average of the third communication latency between computing units in different types of groups in different data centers in multiple data centers of the model training system 200; and the average of the third communication latency between computing units in computing nodes in different data centers in multiple data centers of the model training system 200, etc.

[0155] It is understood that the grouping of computing units in various data centers is based on the type of computing units within each data center. For example, it can be based on the specifications of the computing units, the manufacturers of the computing units, etc., and this application does not impose any restrictions on this.

[0156] The specific methods for measuring communication latency between computing units in different data centers across multiple data centers will be described in detail below, and will not be repeated here.

[0157] S403: Determine the cross-domain parallel training strategy based on communication delay information and training information.

[0158] In some embodiments, the model deployment device 100 determines at least one candidate orchestration domain based on communication latency information and / or training information; and determines a cross-domain parallel training strategy based on the communication latency information, training information, and at least one candidate orchestration domain. The candidate orchestration domain includes a candidate pipeline parallel orchestration domain and / or a candidate data parallel orchestration domain, and the candidate orchestration domain includes at least two data centers and / or multiple computing units within at least two data centers.

[0159] For example, model deployment device 100 determines candidate orchestration domains based on communication latency information, training data distribution information, etc. Cross-domain data parallel orchestration or cross-domain pipelined parallel orchestration is then performed on the candidate orchestration domains to obtain the orchestration result. That is, a cross-domain pipelined parallel training strategy and / or a cross-domain data parallel training strategy between multiple data centers.

[0160] The candidate orchestration domain includes a candidate data parallel orchestration domain and / or a candidate pipeline parallel orchestration domain, and the candidate orchestration domain includes at least two data centers and / or multiple computing units in at least two data centers.

[0161] Assume the communication latency between computing units in every two data centers across multiple data centers (e.g., the measured latency between cross-domain cards) is as shown in Table 1 below.

[0162] The model deployment device 100 can determine candidate pipeline parallel orchestration domains using the following methods.

[0163] For example, the communication latency between computing units in every two data centers in multiple data centers is sorted in ascending order, and the computing units corresponding to the top K communication latencies are assigned to the same candidate pipeline parallel orchestration domain, where K is an integer greater than 1.

[0164] Or, for example, the computing units corresponding to communication latency less than the communication latency threshold in each pair of data centers in multiple data centers can be assigned to the same candidate pipeline parallel orchestration domain.

[0165] For example, the candidate pipeline parallel orchestration domain can be partitioned as follows:

[0166] Multiple data units GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8 in DC1, and multiple data units GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8 in DC2 are assigned to PP candidate orchestration domain 1.

[0167] Multiple data units GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8 in DC3, and multiple data units GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8 in DC5 are assigned to PP candidate orchestration domain 2.

[0168] Table 1. Results of Inter-Domain Card Latency Measurement

[0169] N1 N2 N3 N4 N5 N6 N7 N8 N9 N10 N11 N12 M1 0.64 0.049 0.24 0.678 0.668 0.07 0.904 0.981 0.01 0.882 0.399 0.735 M2 0.573 0.872 0.996 0.519 0.288 0.481 0.948 0.329 0.109 0.987 0.35 0.476 M3 0.237 0.135 0.612 0.956 0.743 0.425 0.987 0.106 0.588 0.922 0.776 0.034 M4 0.26 0.711 0.015 0.708 0.098 0.612 0.796 0.736 0.47 0.536 0.328 0.765 M5 0.757 0.771 0.464 0.761 0.544 0.957 0.255 0.063 0.794 0.556 0.005 0.534 M6 0.344 0.368 0.994 0.707 0.037 0.576 0.498 0.313 0.458 0.087 0.993 0.997 M7 0.521 0.451 0.534 0.054 0.499 0.547 0.137 0.686 0.552 0.697 0.078 0.278 M8 0.45 0.374 0.52 0.936 0.843 0.08 0.009 0.651 0.51 0.298 0.663 0.732 M9 0.572 0.064 0.154 0.353 0.965 0.496 0.729 0.404 0.655 0.669 0.246 0.575 M10 0.595 0.33 0.2 0.007 0.46 0.439 0.71 0.689 0.737 0.963 0.463 0.524 M11 0.717 0.453 0.359 0.962 0.005 0.687 0.802 0.445 0.468 0.017 0.812 0.997 M12 0.093 0.976 0.955 0.993 0.797 0.405 0.167 0.601 0.458 0.423 0.027 0.811

[0170] As shown in Table 1, M1, M2…M12 represent the numbers of each computing unit (e.g., GPU card) in data center DC1, and N1, N2…N12 represent the numbers of each computing unit in data center DC2. The values ​​in Table 1 represent the communication latency between one computing unit in DC1 and another computing unit in DC2. These values ​​can be measured time values ​​or the result of normalizing the measured time values; that is, the communication latency between the two computing units is greater than or equal to 0 and less than or equal to 1.

[0171] For example, the communication latency between M1 and N1 is 0.64, where 0.64 is the result after normalizing the measured time values. The communication latency between M1 and N2 is 0.049, where 0.049 is the result after normalizing the measured time values. This indicates that the communication latency between M1 and N2 is less than the communication latency between M1 and N1. Therefore, when generating candidate pipeline parallel orchestration domains, M1 and N2 can be assigned to the same candidate pipeline parallel orchestration domain.

[0172] It is understood that the above description only uses the example of determining candidate pipeline parallel orchestration domains based on communication latency information to illustrate the process of determining candidate pipeline parallel orchestration domains. In other embodiments, the model deployment device 100 may also determine candidate pipeline parallel orchestration domains based on more training information from the model training system 200, and this application does not impose any limitations on this.

[0173] Assume the training data distribution is as follows: User A: {DC1, DC3}; User B: {DC2, DC3}; User C: {DC4}; User D: {DC1~DC5}; ... Then the model deployment device 100 can assign DC1 and DC3 to one candidate data parallel orchestration domain, assign DC2 and DC3 to another candidate data parallel orchestration domain, and assign DC1, DC2, DC3, DC4, and DC5 to yet another candidate data parallel orchestration domain.

[0174] It is understood that the above description only uses the example of determining the candidate data parallel orchestration domain based on the data distribution information of the training data to illustrate the process of determining the candidate data parallel orchestration domain. In other embodiments, the model deployment device 100 may also determine the candidate data parallel orchestration domain based on more training information and communication latency information of the model training system 200, and this application does not impose any limitations on this.

[0175] In some embodiments, the sum of communication latency between the various computing units corresponding to the cross-domain parallel training strategy is less than a first threshold. That is, during the process of training the model to be trained by the model training system 200 based on the cross-domain parallel training strategy determined by the model deployment device 100, the sum of communication latency between all computing units participating in the model training is less than the first threshold.

[0176] Understandably, in a cross-domain pipeline parallel training strategy, at least two different pipeline stages of the model to be trained are assigned to computing units in different data centers, wherein a pipeline stage includes at least one network layer of the model to be trained.

[0177] Under the cross-domain data parallel training strategy, at least two sub-training data of the training data of the model to be trained are assigned to computing units in different data centers. The sub-training data is a subset of the training data of the model to be trained, and there is no overlap between the sub-training data.

[0178] In some embodiments, a cross-domain pipelined parallel training strategy is used to divide the network layers of the model to be trained into multiple pipeline stages and distribute these multiple pipeline stages to computing units in multiple data centers for parallel training. Furthermore, the cross-domain pipelined parallel training strategy includes at least one of the following parameters: the pipeline stage of the model to be trained executed in each data center, the pipeline partitioning method of the model to be trained (i.e., the cross-domain inter-layer partitioning method), the size of the data processed by each computing node in each data center (e.g., MBS), and the scheduling method of communication operators between computing units in each data center (e.g., configuring the S / R communication operator's stream scheduling scheme as dual-stream scheduling to achieve synchronous data transmission and reception, etc.).

[0179] It is understandable that in existing pipelined parallel technology, the scheduling method of communication operators between various data centers is single-stream scheduling. Data reception and transmission are executed by the same communication operator, and data reception and transmission cannot be executed synchronously, which increases the communication time during model training.

[0180] For example, refer to the above Figure 1B The diagram illustrates the communication time for pipelined parallel training across data centers. GPU3 sends data block 2 to GPU4 during the same clock cycle in which GPU4 receives data block 1. However, after receiving data block 1 from GPU3, GPU4 processes data block 1 first, then sends the processed data block 1 back to GPU3, and then receives data block 2 from GPU3.

[0181] The scheduling mode of the communication operators between the computing units of each data center in the cross-domain pipeline parallel training strategy determined by the model deployment method provided in this application embodiment can be configured as dual-stream scheduling to achieve synchronous data transmission and reception, thereby reducing the communication time during model training and improving model training efficiency.

[0182] It can be understood that the pipeline stages of the training model executed in each data center can include the number of pipeline stages of the training model executed in each data center and the network layer corresponding to the pipeline stage (i.e., cross-domain load).

[0183] In some embodiments, the network deployment device 100 may also determine the pipeline segmentation strategy and pipeline allocation strategy of the model to be trained (i.e., the pipeline stage of the model to be trained executed for each data center) based on virtual pipeline technology. The cross-domain pipeline parallel training strategy may further include the parameter: virtual pipeline pipeline (vPP).

[0184] It is understandable that determining the pipeline segmentation strategy and pipeline allocation strategy for the model to be trained based on virtual pipeline technology can reduce the communication volume between computing units in different data centers by increasing the number of pipeline stages, thereby reducing the latency in the pipeline (e.g., Figure 1B (as shown in the bubble time), thereby improving model training efficiency.

[0185] In some embodiments, a cross-domain data parallel training strategy is used to divide the training data of the model to be trained into multiple sub-training data and distribute the multiple sub-training data to computing units in multiple data centers for parallel training. Furthermore, the cross-domain data parallel training strategy includes at least one of the following parameters: the sub-training data corresponding to each data center, the training data splitting strategy, the communication algorithm between the data centers for gradient aggregation and synchronization, and the parameter configuration information (e.g., timing information, grouping information, etc.) of the communication algorithm between the data centers for gradient aggregation and synchronization.

[0186] Specifically, the model deployment device 100 performs cross-domain data parallel orchestration or cross-domain pipeline parallel orchestration on candidate orchestration domains, and the method for obtaining the orchestration results will be described in detail below, and will not be repeated here.

[0187] S404: Send the cross-domain parallel training strategy to the model training system 200.

[0188] In some embodiments, the model deployment device 100 deploys (sends) the determined cross-domain parallel training strategy to the model training system 200, and the model training system 200 trains the model to be trained based on the cross-domain parallel training strategy determined by the model deployment device 100.

[0189] Specifically, the model training process will be described in detail below, and will not be repeated here.

[0190] Understandable, through Figure 4 The cross-domain parallel training strategy determined by the model deployment method shown can reduce the communication latency of data transmission between computing units in different data centers, thereby reducing model training time and improving model training efficiency.

[0191] The following describes, with reference to the accompanying drawings, a method for determining communication latency information between computing units in different data centers within multiple data centers of the model training system 200 using the model deployment device 100.

[0192] In some embodiments, the model deployment device 100 measures the communication latency between computing units in different data centers within the model training system 200 based on network performance testing tools (such as iperf, IFIT, etc.).

[0193] For example, Figure 5 A flowchart illustrating a communication delay measurement method is shown according to an embodiment of this application. It can be understood that... Figure 5 The process shown is executed by the model deployment device 100. For simplicity, the following description... Figure 5 The execution entity will not be described again when the process is shown.

[0194] like Figure 5 As shown, this process includes, but is not limited to:

[0195] S501: Obtain the interconnection relationship between cards.

[0196] In some embodiments, the model deployment device 100 obtains the topology information between different computing units in multiple data centers of the model training system 200, i.e., the card interconnection relationship, through network controllers, user configurations, etc.

[0197] S502: Measure point-to-point delay using iperf / IFIT and other methods.

[0198] In some embodiments, the model deployment device 100 uses network performance measurement technology (such as iperf, IFIT and other network performance testing tools) to obtain the communication latency between any two computing units belonging to different data centers in the model training system 200, that is, the point-to-point latency.

[0199] In other embodiments, the communication latency between any two computing units belonging to different data centers can be determined by taking multiple measurements and averaging them. This provides a more accurate communication latency result compared to a single measurement.

[0200] S503: Generate point-to-point communication delay matrix.

[0201] In some embodiments, the model deployment device 100 generates a communication delay matrix based on the acquired communication delay.

[0202] For example, the model deployment device 100 divides the computing units in multiple data centers of the model training system 200 into N measurement groups. Each measurement group includes two computing units belonging to different data centers, where N is an integer greater than 1. Network performance measurement technology is used to obtain the communication latency between the computing units in each measurement group (as an example of the first communication latency), and a communication latency matrix is ​​generated based on the obtained communication latency (as an example of the first communication latency matrix). That is, S502 is repeated N times, and after each measurement, the generated communication latency matrix is ​​updated based on the measurement results.

[0203] It can be understood that the communication delay matrix includes the communication delay between various computing units in different data centers across multiple data centers. For example, the communication delay matrix can be found in Table 1.

[0204] S504: Output result.

[0205] In some embodiments, after the model deployment device 100 completes the measurement of the communication latency between all computing units located in different data centers in multiple data centers of the model training system 200, it outputs the results, such as a communication latency matrix.

[0206] In other embodiments, the model deployment device 100 adds relevant fields to the communication operators corresponding to the parallel training strategy (e.g., pipelined parallel S / R communication operators, data parallel AR communication operators), and measures the communication latency between computing units in different data centers in the model training system 200 based on the communication operators.

[0207] For example, Figure 6 A flowchart illustrating another communication delay measurement method is shown according to an embodiment of this application. It can be understood that... Figure 6 The process shown is executed by the model deployment device 100. For simplicity, the following description... Figure 6 The execution entity will not be described again when the process is shown.

[0208] like Figure 6 As shown, this process includes, but is not limited to:

[0209] S601: Construct S / R communication operators and / or AR communication operators.

[0210] In some embodiments, the model deployment device 100 adds relevant fields to the pipelined parallel S / R communication operator to construct an S / R communication operator with communication delay measurement function.

[0211] In other embodiments, the model deployment device 100 adds relevant fields to the data-parallel AR communication operator to construct an AR communication operator with communication latency measurement function.

[0212] S602: Compile the constructed S / R communication operator and / or AR communication operator.

[0213] In some embodiments, after constructing the S / R communication operator and / or AR communication operator, the model deployment device 100 sends the constructed S / R communication operator and / or AR communication operator to the compiler for compilation and execution, and obtains the communication latency between any two computing units belonging to different data centers in the model training system 200.

[0214] In other embodiments, the communication latency between any two computing units belonging to different data centers can be determined by taking multiple measurements and averaging them. This provides a more accurate communication latency result compared to a single measurement.

[0215] S603: Generate point-to-point communication delay matrix.

[0216] In some embodiments, the model deployment device 100 generates a communication delay matrix based on the acquired communication delay.

[0217] For example, the model deployment device 100 divides the computing units in multiple data centers into N measurement groups, each measurement group including two computing units belonging to different data centers. Based on communication operators, the communication latency between the computing units in each measurement group is obtained, and a communication latency matrix is ​​generated based on the obtained communication latency. That is, S502 is repeated N times, and after each measurement, the generated communication latency matrix is ​​updated based on the measurement results.

[0218] It can be understood that the communication delay matrix includes the communication delay between various computing units in different data centers across multiple data centers. For example, the communication delay matrix can be found in Table 1.

[0219] S604: Output result.

[0220] In some embodiments, after the model deployment device 100 completes the measurement of the communication latency between all computing units located in different data centers in multiple data centers of the model training system 200, it outputs the results, such as a communication latency matrix.

[0221] Understandable. Figure 6 The measurement method shown measures the communication delay between computing units during data transmission by the communication operator. Compared to... Figure 5 The method shown is to measure using network performance measurement techniques. Figure 6 The measurement method shown yields more accurate results.

[0222] It is understood that the number of computing units is usually large in cross-domain training scenarios. In some embodiments, the model deployment device 100 may also select multiple computing units from each data center in the model training system 200 as computing unit samples for each data center. Then, based on the communication operator with communication latency measurement function (e.g., S / R communication operator, AR communication operator, etc.), the communication latency between each computing unit in different computing unit samples is measured (as an example of the second communication latency), and the average value of the communication latency between each computing unit in different computing unit samples is taken as the communication latency between computing units in different data centers. In this way, the measurement time and measurement cost of communication latency can be shortened. It is understood that in other alternative embodiments, the mode, median, etc. of the communication latency between each computing unit in different computing unit samples can also be taken as the communication latency between computing units in different data centers, and this application does not limit this.

[0223] In some embodiments, each data center includes different types of data units. The model deployment device 100 can divide the computing units in multiple data centers into multiple type groups based on the type of computing unit. Then, based on the communication operator with communication latency measurement function described above, the communication latency between computing units in each type group of different data centers is obtained (as an example of a third communication latency). The average value of the communication latency between multiple computing units in each type group of different data centers is taken as the communication latency between computing units in each type group of different data centers. It is understood that in other alternative embodiments, the mode, median, etc. of the communication latency between multiple computing units in each type group of different data centers can also be taken as the communication latency between computing units in different data centers, and this application does not limit this.

[0224] In other embodiments, each data center includes at least one computing node, and multiple computing units in each data center correspond to at least one computing node in each data center. That is, it includes multiple computing units, and multiple computing units correspond to at least one computing node. The model deployment device 100 can measure the communication latency between computing units in each computing node located in different data centers based on the communication operator with the communication latency measurement function described above (as an example of the fourth communication latency). Then, the average value of the communication latency between computing units in each computing node of different data centers is taken as the communication latency between computing units in each computing node of different data centers. It is understood that in other alternative embodiments, the mode, median, etc. of the communication latency between computing units in each computing node of different data centers can also be taken as the communication latency between computing units in different data centers, and this application does not limit this.

[0225] It is understood that the above description only takes the acquisition of communication latency based on a communication operator with communication latency measurement function as an example. In other embodiments, the above methods of sampling, type-based grouping, and computing node measurement can also measure the communication latency between computing units through network performance measurement technology.

[0226] It is understandable that the above methods of sampling, type-based grouping, and computation node-based measurement can also be combined.

[0227] For example, the model deployment device 100 can divide the computing units within each computing node of multiple data centers into multiple type groups based on the type of computing unit. Then, based on the communication operator with communication latency measurement function mentioned above, the communication latency between computing units in each type group of different data centers is obtained. The average value of the communication latency between multiple computing units in each type group of different data centers is taken as the communication latency between computing units in each type group of different data centers.

[0228] For example, the model deployment device 100 can select multiple computing units from each computing node in each data center, as computing unit samples for each computing node in each data center. Then, based on the communication operator with communication latency measurement function, the communication latency between each computing unit in the computing unit samples of each computing node in different data centers is measured. The average communication latency between each computing unit in different computing unit samples is taken as the communication latency between computing units in each computing node of different data centers.

[0229] This application does not limit the specific method for obtaining the communication latency between computing units in different data centers.

[0230] The following describes, with reference to the accompanying diagram, a method for the model deployment device 100 to perform cross-domain data parallel orchestration or cross-domain pipeline parallel orchestration on candidate orchestration domains to obtain orchestration results.

[0231] For example, Figure 7 A flowchart illustrating a candidate domain orchestration method is shown according to an embodiment of this application. It can be understood that... Figure 7 The process shown is executed by the model deployment device 100. For simplicity, the following description... Figure 7 The execution entity will not be described again when the process is shown.

[0232] like Figure 7 As shown, this process includes, but is not limited to:

[0233] S701: Obtain model information, topology information, and candidate orchestration domains.

[0234] In some embodiments, the model deployment device 100 acquires model information (e.g., number of model layers, model parameters, etc.), topology information (e.g., topology information between multiple data centers, topology information between multiple computing nodes within each data center, topology information between multiple computing units within each data center, etc.), and candidate orchestration domains (e.g., candidate pipeline parallel orchestration domain, candidate data parallel orchestration domain, etc.) of the model to be trained.

[0235] The candidate orchestration domains can be referenced from the above. Figure 4The method for determining candidate arrangement domains in S403 is shown. This application will not elaborate on this method.

[0236] S702: Perform cross-domain data parallel orchestration.

[0237] In some embodiments, the model deployment device 100 determines a cross-domain data parallel training strategy between different data centers based on model information, topology information, and candidate data parallel orchestration domains. For example, based on data center information / computing unit information in each candidate data parallel orchestration domain, and the amount of training data, it determines a data parallel partitioning method (e.g., partitioning of training data, allocation of training data, etc.), a communication algorithm for gradient aggregation and synchronization between different data centers, and the configuration of the communication algorithm for gradient aggregation and synchronization between different data centers (configuration information of parameters such as timing and grouping).

[0238] The communication algorithms include, but are not limited to, the ring algorithm, the NHR algorithm, and the HD algorithm.

[0239] S703: Performs intra-domain tensor parallel / model parallel / expert parallel orchestration.

[0240] In some embodiments, the model deployment device 100 performs one or more parallel orchestrations of tensor parallelism (TP), sequential parallelism (SP), and expert parallelism (EP) within each data center to obtain an intra-domain parallel training strategy.

[0241] Tensor parallelism distributes the model parameters and computational tasks of a single or multiple layers of the model to be trained across different computational units within various data centers. For example, the model parameters of each layer of a neural network model can be assigned to different computational units. Sequence parallelism distributes different parts of the model to be trained (e.g., different layers or subnetworks) across different computational units within various data centers. Expert parallelism distributes the expert networks in the model to be trained across different computational units within various data centers. Each expert network consists of multiple smaller models, each processing a specific portion of the input data.

[0242] S704: Perform cross-domain micro-batch size / virtual pipeline parallel pipeline orchestration.

[0243] In some embodiments, the model deployment device 100 determines the cross-domain pipeline parallel training strategy between different data centers based on model information, topology information, cross-domain data parallel training strategy, intra-domain parallel training strategy, and candidate pipeline parallel orchestration domains.

[0244] For example, the model deployment device 100 calculates the pipeline parallel communication volume according to the aforementioned cross-domain data parallel training strategy and intra-domain parallel training strategy, and determines the cross-domain pipeline parallel training strategy between different data centers based on the pipeline parallel communication volume, model information, topology information and candidate pipeline parallel orchestration domains.

[0245] Among them, the domain pipeline parallel training strategy includes, but is not limited to, MBS, vPP, inter-layer splitting method, cross-domain load balancing strategy, cross-domain communication operator configuration (i.e., the scheduling method of communication operators between computing units in each data center, for example, configuring the S / R communication operator's stream scheduling scheme as dual-stream scheduling to achieve synchronous data transmission and reception, etc.).

[0246] S705: Determines whether end-to-end cavitation / communication exposure time / computational efficiency loss is minimized.

[0247] The model deployment device 100 statistically analyzes one or more of the following: end-to-end bubbling time for pipeline parallelism, communication exposure time for pipeline parallelism and / or data parallelism, and computational efficiency loss of the communication algorithm. If the statistically calculated end-to-end bubbling time / communication exposure time / computational efficiency loss is less than a set threshold, it determines whether the end-to-end bubbling time / communication exposure time / computational efficiency loss is minimized. For example, if the model deployment device 100 detects that the sum of communication delays between various computing units during data training is less than a set communication delay threshold, it determines that the end-to-end bubbling time / communication exposure time / computational efficiency loss is minimized.

[0248] In some embodiments, if the determination is yes, it means that based on the current cross-domain data parallel training strategy, intra-domain parallel training strategy, and cross-domain pipeline parallel training strategy, one or more of the end-to-end cavitation time of pipeline parallelism, communication exposure time of pipeline parallelism and / or data parallelism, and computational efficiency loss of communication algorithm meet the output conditions. Then the model deployment device 100 can execute S706 and output the intra-domain-inter-domain orchestration result.

[0249] In other embodiments, if the determination result is negative, it indicates that based on the current cross-domain data parallel training strategy, intra-domain parallel training strategy, and cross-domain pipelined parallel training strategy, one or more of the following—end-to-end cavitation time of pipelined parallelism, communication exposure time of pipelined parallelism and / or data parallelism, and computational efficiency loss of communication algorithms—do not meet the output conditions. Therefore, the model deployment device 100 needs to adjust at least one of the cross-domain data parallel training strategy, intra-domain parallel training strategy, and cross-domain pipelined parallel training strategy. The model deployment device 100 executes S702 to perform cross-domain data parallel orchestration.

[0250] S706: Output intra-domain and inter-domain orchestration results.

[0251] In some embodiments, after the model deployment device 100 determines that one or more of the following conditions are met based on the current cross-domain data parallel training strategy, intra-domain parallel training strategy, and cross-domain pipelined parallel training strategy: the end-to-end cavitation time of pipelined parallelism, the communication exposure time of pipelined parallelism and / or data parallelism, and the computational efficiency loss of the communication algorithm, the model deployment device 100 outputs the current cross-domain data parallel training strategy, intra-domain parallel training strategy, and cross-domain pipelined parallel training strategy, i.e., the intra-domain-inter-domain orchestration result.

[0252] The structure and function of the software module of the model deployment device provided in the embodiments of this application are described below with reference to the accompanying drawings.

[0253] For example, Figure 8 A schematic diagram of the structure of a model deployment device 100 is shown according to an embodiment of this application.

[0254] like Figure 8 As shown, the model deployment device 100 includes: an information acquisition module 110, a parallel orchestration module 120, and a training module 130.

[0255] The information acquisition module 110 is used to acquire training information from the model training system 100. This training information includes, but is not limited to: topology information of multiple data centers, user-configured training parameters, data distribution information of the training data, and model information of the model to be trained.

[0256] The information acquisition module 110 is also used to determine communication latency information between computing units in different data centers within multiple data centers. The method for determining the communication latency information between computing units in different data centers within multiple data centers can refer to the above-described method. Figures 4 to 6 The relevant descriptions regarding the acquisition of communication delay information between computing units are not repeated here.

[0257] The parallel orchestration module 120 is used to determine cross-domain parallel training strategies based on communication latency information and training information. These cross-domain parallel training strategies include cross-domain pipeline parallel training strategies and / or cross-domain data parallel training strategies.

[0258] In some embodiments, the parallel orchestration module 120 is further configured to determine at least one candidate orchestration domain based on communication latency information and / or training information, wherein the candidate orchestration domain includes a candidate pipeline parallel orchestration domain and / or a candidate data parallel orchestration domain, and the candidate orchestration domain includes at least two data centers and / or multiple computing units in at least two data centers. Then, a cross-domain parallel training strategy is determined based on the communication latency information, training information, and at least one candidate orchestration domain.

[0259] Understandably, in a cross-domain pipelined parallel training strategy, at least two different pipeline stages of the model to be trained are allocated to computing units in different data centers. Similarly, in a cross-domain data parallel training strategy, at least two sub-training data sets of the training data of the model to be trained are allocated to computing units in different data centers, where the sub-training data sets are subsets of the training data of the model to be trained.

[0260] In some embodiments, the communication latency between the various computing units corresponding to the cross-domain parallel training strategy is less than a first threshold.

[0261] Training module 130 is used to send cross-domain parallel training strategies to model training system 200.

[0262] It should be noted that the implementation of each module can also be referenced accordingly. Figure 4 The corresponding description of the method embodiment shown describes the methods and functions performed by the model training system 100 in the above embodiments.

[0263] The following section, with reference to the accompanying diagram, describes the cross-domain parallel training strategy determined by the model deployment device in the model training system, and introduces the process of training the model to be trained.

[0264] Figure 9 A schematic flowchart of a model training method is shown according to an embodiment of this application. It can be understood that... Figure 9 The execution entity of the illustrated process is the model training system 200, which includes multiple data centers, each of which includes at least one computing unit. For simplicity, the following description... Figure 9 The execution entity will not be described again when the process is shown.

[0265] like Figure 9 As shown, this process includes, but is not limited to:

[0266] S901: Obtain cross-domain parallel training strategy from model deployment device 100.

[0267] In some embodiments, a data center includes at least one computing node (e.g., a server), and a computing node includes at least one computing unit. That is, multiple computing units in a data center correspond to at least one computing node.

[0268] In some embodiments, the cross-domain parallel training strategy is determined by the model deployment device 100 based on the communication latency information between computing units in different data centers of the multiple data centers of the model training system 200 and the training information of the model training system 200.

[0269] Specifically, the content and determination method of the cross-domain parallel training strategy can be referred to the relevant descriptions of the aforementioned model deployment methods, which will not be repeated here.

[0270] S902: The cross-domain parallel training strategy is used to perform multiple rounds of iterative training on the model to be trained.

[0271] In some embodiments, the model training system 200 performs multiple rounds of iterative training on the model to be trained based on a cross-domain parallel training strategy.

[0272] It is understandable that communication may be unstable in cross-domain training scenarios due to factors such as the distance between different data centers.

[0273] Therefore, in some embodiments, when the sum of the communication delays between the various computing units corresponding to the cross-domain parallel training strategy is greater than or equal to a first threshold, the model training system 200 adjusts the cross-domain parallel training strategy so that the sum of the communication delays between the various computing units corresponding to the adjusted cross-domain parallel training strategy is less than the first threshold.

[0274] For example, the model training system 100 can monitor communication during training, such as monitoring the communication latency of pipelined parallel S / R communication operators and data-parallel AR communication operators. If the sum of the communication latency between the various computing units corresponding to the cross-domain parallel training strategy is less than a first threshold, the cross-domain parallel training strategy is adjusted and the training configuration information is updated. The updated training configuration information is then used for training in the next round of training.

[0275] The structure of the model training system involved in the embodiments of this application is described below with reference to the accompanying drawings.

[0276] For example, Figure 10 A schematic diagram of the structure of a model training system 200 is shown according to an embodiment of this application.

[0277] like Figure 10 As shown, the model training system 200 includes multiple data centers (data center 1, ..., data center L), where L is an integer greater than 1. Each data center includes multiple computing nodes (computing node 1, ..., computing node P), where P is an integer greater than 1.

[0278] In some embodiments, a computing node may be a server. Each computing node includes multiple computing units (computing unit 1, ..., computing unit Q), where Q is an integer greater than 1. Each computing node may also include memory, communication interfaces, and buses, etc.

[0279] In some embodiments, the computing unit includes, but is not limited to, GPU, NPU, TPU, microprocessor, application-specific integrated circuit, etc., for executing relevant programs to achieve the functions required by the computing unit in the model training system 200 of this application embodiment.

[0280] In some embodiments, the model training system 200 further includes a control node (not shown in the figure), which is used to obtain a model training strategy (e.g., a cross-domain parallel training strategy) and control the model training system 200 to train the model to be trained based on the model training strategy.

[0281] In other embodiments, the control node is also used to determine the model training strategy (e.g., cross-domain parallel training strategy), that is, the control node is used to implement the functions implemented by the model deployment device 100 described above.

[0282] It is understood that the model deployment device 100 can be a standalone device or a control node in the model training system 200 used to determine the model training strategy. This application does not impose any restrictions on this.

[0283] The hardware structure of the model deployment device 100 is described below with reference to the accompanying drawings.

[0284] For example, Figure 11 A schematic diagram of the hardware structure of a model deployment device 100 is shown according to an embodiment of this application.

[0285] like Figure 11 As shown, the model deployment device 100 includes one or more (only one is shown in the figure) processing units 1110, memory 1120, communication interface 1130, and bus 1140. The processing units 1110, memory 1120, and communication interface 1130 are interconnected via bus 1140.

[0286] The processing unit 1110 includes, but is not limited to, a central processing unit (CPU), GPU, NPU, TPU, microprocessor, application-specific integrated circuit, etc., for executing relevant programs to achieve the functions required by the model deployment device 100 of this application embodiment.

[0287] Memory 1120 may include one or more memories for storing data or one or more applications. The memory may be read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM).

[0288] The processing unit 1110 can also be an integrated circuit chip with signal processing capabilities. The aforementioned processing unit 1110 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1120. The processing unit 1110 reads the information in memory 1120 and, in conjunction with its hardware, completes the functions required by the model deployment device 100 in the embodiments of this application.

[0289] The communication interface 1130 is used to enable communication between the model deployment device 100 and other devices or communication networks. In some embodiments, the model deployment device 100 establishes a communication connection with the model training system 200 through the communication interface 1130 and sends a cross-domain parallel training strategy to the model training system 200.

[0290] Bus 1140 is used to connect processing unit 1110, memory 1120, communication interface 1130 and other possible modules or circuits.

[0291] It should be understood that Figure 11 The structure of the model deployment device 100 shown is only an example. In other embodiments, the model deployment device 100 may include more or fewer modules, which is not limited here.

[0292] It is understood that the model deployment device 100 can be a standalone device or a control node in the model training system 200 used to determine the model training strategy. This application does not impose any restrictions on this.

[0293] The following example uses model deployment device 100 as the control node in model training system 200, combined with... Figure 10 This application provides a detailed description of the model deployment and model training methods provided in its embodiments. Specifically, the control node in the model training system 200 determines a cross-domain parallel training strategy (implementing the function of the aforementioned model deployment device 100), and completes the training of the model to be trained based on the determined domain parallel training strategy.

[0294] For example, Figure 12 A logical block diagram of a model deployment and model training process is shown according to an embodiment of this application.

[0295] like Figure 12 As shown, basic information is input into the model training system 200. This basic information (or training information) includes model information of the model to be trained, topology information of the model training system 200, and data distribution information of the data used for training. The data distribution information of the data used for training may include the geographical distribution information of the data (e.g., information about the corresponding data centers).

[0296] After acquiring the input basic information, the model training system 200 performs cross-DC communication sensing and orchestration domain generation. This cross-DC communication sensing and orchestration domain generation includes DC communication delay detection and orchestration domain generation.

[0297] DC communication latency detection includes measuring cross-DC communication latency, i.e., the communication latency between various computing units in different data centers, through S / R communication operators, AR communication operators, or network measurement schemes (such as IFIT).

[0298] The orchestration domain generation includes constructing candidate orchestration domains based on information such as data center distribution and user training data distribution.

[0299] After determining the candidate orchestration domains, the model training system 200 performs automatic parallel orchestration of the bounded domain and distance-aware methods. The automatic parallel orchestration of the bounded domain and distance-aware methods includes bounded domain DP orchestration and cross-domain PP orchestration.

[0300] Limited-domain DP orchestration includes: outputting cross-domain DP orchestration strategies and AR communication algorithms, that is, determining cross-domain data parallel training strategies.

[0301] Cross-domain PP orchestration includes: based on the communication latency, model information, orchestration MBS, vPP and other information of the S / R communication operators across data centers, realizing computational communication masking, and outputting cross-domain PP orchestration strategy, that is, determining the cross-domain pipeline parallel orchestration strategy.

[0302] After completing the automatic parallel orchestration of the limited domain and distance awareness, the model training system 200 distributes and compiles the cross-domain training orchestration results. The cross-domain orchestration results include information such as parallel training strategies, PP splitting strategies, MBS, and vPP.

[0303] It is understood that the process of determining the cross-domain parallel training strategy is implemented by the control node (not shown in the figure) of the model training system 200, and the following model training process is implemented by the computing nodes in at least two data centers in the model training system 200.

[0304] Specifically, the model training system 200 begins training after compilation and dynamically adjusts itself during runtime. For example, during model training, communication performance is monitored, and orchestration is dynamically optimized based on the monitoring results.

[0305] For example, the model training system 200 detects the communication latency of the S / R communication operator in pipeline parallelism (PP), the communication latency of the AR communication operator in data parallelism (DP), and the masking performance during model runtime. It issues an alarm when network congestion or degradation exceeds a threshold (e.g., the sum of communication latency exceeds a first threshold) occurs, and dynamically adjusts the orchestration strategy of data parallelism and / or pipeline parallelism based on real-time monitoring results, and updates the training configuration information.

[0306] The following is combined with Figure 13 This application describes in detail the process of automatic parallel orchestration of the control node (or model deployment device 100) of the model training system 200 in the model deployment method provided in the embodiments of this application.

[0307] For example, Figure 13 An embodiment of this application illustrates a logic block diagram for automatic parallel orchestration.

[0308] like Figure 13 As shown, the control node of the model training system 200 obtains model information of the model to be trained through the model information interface, topology information and physical environment information through the topology and physical environment information interface, and obtains point-to-point latency detection information. It can be understood that the point-to-point latency detection information can refer to the communication latency between the various computing units shown in Table 1 above.

[0309] The topology and physical environment information can include: scale: 192 cards; distance: 300 kilometers across cities; network topology: OTN + router + switch; convergence ratio: 4:1; budget: 18 million+ etc.

[0310] The control nodes of the model training system 200 automatically perform parallel orchestration based on the model information, topology information, physical environment information of the model to be trained, and point-to-point delay detection information.

[0311] For example, determine the pipeline parallel training strategy: perform 3D sequence parallel optimization to optimize memory usage and communication time in a coordinated manner; perform virtual pipeline parallel optimization and folding arrangement parallel optimization to reduce cavitation time and improve utilization.

[0312] For example, determine the data parallel training strategy: perform expert parallelism (e.g., sparse model parallel optimization) to reduce communication exposure time and improve the training performance of mixture of experts (MoE); perform data parallel communication masking optimization to improve linearity.

[0313] For example, determine the tensor parallel training strategy: optimize memory based on memory management strategies (such as the ZeRo strategy) to reduce GPU memory usage and improve utilization.

[0314] The orchestration results of automatic parallel orchestration can be: outputting cross-domain pipeline parallel configurations (such as MBS, vPP, cross-domain inter-layer partitioning methods, cross-domain load balancing strategies, etc.) and cross-domain communication operator configurations (such as S / R communication operator flow scheduling schemes); constructing data parallel modeling and simulation based on candidate orchestration domains, network topology, and convergence ratios, and outputting high-performance data parallel orchestration results, such as training data partitioning strategies, communication algorithms, and communication algorithm configurations (timing, grouping, etc.).

[0315] In some embodiments, this application also provides a computer-readable storage medium storing at least one computer program instruction, at least one program segment, code set, or instruction set, which is loaded and executed by a model deployment device to implement the model deployment method provided in the above-described method embodiments.

[0316] In some embodiments, this application also provides a computer program product, which includes computer program instructions that, when executed by a model deployment device, enable the model deployment device to implement the model deployment methods provided in the above-described method embodiments.

[0317] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0318] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0319] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0320] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0321] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0322] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0323] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0324] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A model deployment method, characterized in that, The method includes: Acquire training information of a model training system, wherein the model training system is used to train a model to be trained, and the model training system includes multiple data centers, wherein each data center includes at least one computing unit, and the training information includes at least one of the topology information of the multiple data centers and the data distribution information of the training data of the model to be trained; Determine the communication latency information between computing units in different data centers among the multiple data centers; Based on the communication latency information and the training information, a cross-domain parallel training strategy is determined, wherein, under the cross-domain parallel training strategy, computing units in at least two data centers in the model training system execute the training of the model to be trained. The cross-domain parallel training strategy is sent to the model training system.

2. The method according to claim 1, characterized in that, Determining the communication latency information between computing units in different data centers among the multiple data centers includes: Based on network performance testing tools, the communication latency between computing units in different data centers is measured; or, The communication latency between computing units in different data centers is obtained based on a communication operator, wherein the communication operator includes a field for measuring the communication latency between computing units.

3. The method according to claim 1, characterized in that, The communication latency information between computing units in different data centers within the multiple data centers includes: The first communication delay between various computing units located in different data centers of the plurality of data centers.

4. The method according to claim 1, characterized in that, The communication latency information between computing units in different data centers within the multiple data centers includes: The average value of the second communication latency between each computing unit in each computing unit sample located in different data centers of the plurality of data centers.

5. The method according to claim 1, characterized in that, The communication latency information between computing units in different data centers within the multiple data centers includes: The average third communication latency between computing units in various type groups located in different data centers of the plurality of data centers, wherein the type groups are based on the type of computing units in each data center.

6. The method according to any one of claims 1 to 5, characterized in that, The cross-domain parallel training strategy includes at least one of the following: A cross-domain pipelined parallel training strategy, wherein, under the cross-domain pipelined parallel training strategy, at least two different pipeline stages of the model to be trained are assigned to computing units in different data centers, wherein, a pipeline stage includes at least one network layer of the model to be trained. A cross-domain data parallel training strategy, wherein at least two sub-training data of the training data are allocated to computing units in different data centers, wherein the sub-training data is a subset of the training data of the model to be trained, and there is no overlap between the sub-training data.

7. The method according to claim 6, characterized in that, The step of determining a cross-domain parallel training strategy based on the communication latency information and the training information includes: Based on the communication latency information and / or the training information, at least one candidate orchestration domain is determined, wherein the candidate orchestration domain includes a candidate pipeline parallel orchestration domain and / or a candidate data parallel orchestration domain, and the candidate orchestration domain includes at least two data centers and / or multiple computing units in at least two data centers; The cross-domain parallel training strategy is determined based on the communication latency information, the training information, and the at least one candidate orchestration domain.

8. The method according to claim 7, characterized in that, The communication latency between computing units located in different data centers within the candidate pipeline parallel orchestration domain is less than the communication latency threshold.

9. The method according to claim 7, characterized in that, The candidate data parallel orchestration domain corresponds to different data centers that contain training data from the same users.

10. The method according to claim 7, characterized in that, The cross-domain pipelined parallel training strategy includes at least one of the following parameters: The pipeline stages of the model to be trained executed for each data center, the pipeline segmentation method of the model to be trained, the size of the data processed by each computing unit in each data center, and the scheduling method of communication operators between data centers; or, The cross-domain data parallel training strategy includes at least one of the following parameters: The sub-training data corresponding to each data center, the splitting strategy of the training data, the communication algorithm for gradient aggregation and synchronization between the data centers, and the parameter configuration information of the communication algorithm for gradient aggregation and synchronization between the data centers.

11. The method according to claim 1, characterized in that, The topology information of the multiple data centers includes at least one of the following: The distances between the multiple data centers, the topology between the multiple data centers, and the topology between multiple computing units within each of the multiple data centers; or, The data distribution information of the training data includes at least one of the following: The training data corresponds to a distributed data center, and the amount of training data distributed in each data center is as follows.

12. A model training method, characterized in that, The method is applied to a model training system comprising multiple data centers, wherein each data center includes at least one computing unit; and the method includes: A cross-domain parallel training strategy is obtained from the model deployment device, wherein the cross-domain parallel training strategy is determined by the model deployment device based on communication latency information between computing units in different data centers of the multiple data centers and training information of the model training system. The training information includes at least one of the topology information of the multiple data centers and data distribution information of the training data. Under the cross-domain parallel training strategy, computing units in at least two data centers in the model training system perform training on the model to be trained. The cross-domain parallel training strategy is used to perform multiple rounds of iterative training on the model to be trained.

13. The method according to claim 12, characterized in that, The method further includes: During the training process, the sum of communication delays between each computing unit corresponding to the cross-domain parallel training strategy is determined; If the sum of the communication delays between the computing units corresponding to the cross-domain parallel training strategy is greater than or equal to a first threshold, the cross-domain parallel training strategy is adjusted so that the sum of the communication delays between the computing units corresponding to the adjusted cross-domain parallel training strategy is less than the first threshold.

14. A model deployment device, characterized in that, include: An information acquisition module is used to acquire training information of a model training system, wherein the model training system is used to train a model to be trained, and the model training system includes multiple data centers, wherein each data center includes at least one computing unit, and the training information includes at least one of the topology information of the multiple data centers and the data distribution information of the training data of the model to be trained; The information acquisition module is also used to determine the communication latency information between computing units in different data centers among the plurality of data centers; A parallel orchestration module is used to determine a cross-domain parallel training strategy based on the communication latency information and the training information, wherein... Under the cross-domain parallel training strategy, computing units in at least two data centers in the model training system perform training on the model to be trained. The training module is used to send the cross-domain parallel training strategy to the model training system.

15. A model deployment device, characterized in that, include: A memory for storing instructions executed by one or more processing units of the model deployment device; as well as The processing unit is one of the processing units of the model deployment device, and is used to execute the instructions stored in the memory to implement the method of any one of claims 1 to 11.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed on the system, cause the model deployment device to implement the method of any one of claims 1 to 11.

Images