A computing power network communication efficiency optimization method and device for large model training

Abstract

The present application relates to the technical field of computing power network communication, and specifically provides a computing power network communication efficiency optimization method and device for large model training, comprising: calculating the shortest path from a given data center to all remaining data centers based on the bidirectional transmission delay between data centers; distributing the network layers to be trained to each data center based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained; controlling the parallel training of each data center on the network layer divided by it in a pipeline slicing manner, and performing load balancing after the training of each batch of samples is completed until the training of the large model to be trained is completed. The technical scheme provided by the present application effectively optimizes the communication efficiency of the computing power network during large model training, improves the computing / communication ratio, thereby improving the resource utilization rate of the computing power network and accelerating the model training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing power network communication, in particular to a computing power network communication efficiency optimization method and device for large model training. BACKGROUND

[0002] In recent years, with the continuous improvement and progress of deep learning (DL) technology, artificial intelligence has been widely applied in many fields, such as computer vision, natural language processing, autonomous driving and speech recognition, etc. However, traditional small models are increasingly unable to keep up with the trend of the times, and therefore a large number of researchers continue to expand the size of the model and the dataset, which has brought about problems such as significant extension of model training time and insufficient memory space. Although hardware manufacturers are constantly improving the performance of computing chips, it is still not enough to directly solve these problems from the hardware level. In order to speed up the training of models and support the training of larger models, the academic and industrial communities usually use Distributed Deep Learning (DDL) technology to train large models, that is, multiple GPUs or even multiple GPU servers are used to train in parallel within a GPU cluster / data center. Data centers have to continuously expand their scale to accommodate the growing number of model parameters. For the training of super-large models, such as GPT-3 with 175 billion parameters and PaLM with 540 billion parameters, a data center may need thousands or even tens of thousands of GPUs to complete the training within an acceptable time.

[0003] It is foreseeable that the size of the model will continue to grow, and it will be difficult for a single data center to support such a scale of training. Therefore, computing power networks are proposed. As a new type of network architecture, computing power networks not only face the major needs of China's current digital economic development, but also aim to build a self-controllable, technologically leading, and functionally integrated system, to achieve intelligent, efficient and flexible computing power resource integration and scheduling, to meet the differentiated computing power service needs of industries, and to provide support for the development of computing power and intelligent industries. Although computing power networks are based on traditional computer networks, how to achieve efficient communication between nodes to meet the training needs of artificial intelligence large models is still a key problem.

[0004] In the computing power network, artificial intelligence large models are usually trained in parallel by multiple data centers in a model-parallel manner, and the huge original sample data and a large amount of intermediate data generated during the training process will generate huge communication overhead when transmitted between data centers. When the communication time that does not overlap with the calculation time accounts for a high proportion in the total time of training, it will seriously affect the efficiency of model training. If the traditional communication method is used to transmit data between data centers, problems such as too large transmission delay and a large number of GPUs idle for waiting for data may occur. If a new efficient communication protocol is designed for this purpose, it is not only difficult, but also difficult to implement in the short term; and the method of upgrading the transmission medium to obtain greater bandwidth depends on a major breakthrough in materials science. Therefore, a feasible method is needed to reduce the communication overhead by optimizing the data transmission method on the basis of traditional transmission media and communication protocols. SUMMARY

[0005] In order to overcome the above defects, the application provides a computing power network communication efficiency optimization method and device for large model training.

[0006] In a first aspect, a computing power network communication efficiency optimization method for large model training is provided, which comprises:

[0007] Based on the bidirectional transmission delay between data centers, the shortest path from a given data center to all remaining data centers is calculated.

[0008] Based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained, each data center is assigned a network layer to be trained.

[0009] The pipeline slicing method is used to control the parallel training of each data center on the network layer divided by it, and load balancing is performed after the training of each batch of samples is completed until the training of the large model to be trained is completed.

[0010] Preferably, based on the bidirectional transmission delay between data centers, the shortest path from a given data center to all remaining data centers is calculated, which comprises:

[0011] A transmission delay adjacency matrix is constructed.

[0012] A given data center is used as the starting point of the path, and a backtracking method is used to traverse the path constructed by data centers in different orders to record the corresponding path and total transmission delay.

[0013] The total transmission delays of all paths are compared, and the path with the smallest total transmission delay is taken as the shortest path.

[0014] Preferably, the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained are used to allocate the network layers to be trained to each data center, including:

[0015] The constraint condition is constructed based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained;

[0016] The network layers to be trained are allocated to each data center with the goal of meeting the constraint condition.

[0017] Further, the constraint condition is as follows:

[0018]

[0019] In the above formula, layers j is the number of network layers allocated to the jth data center, Cal k is the computing amount of the network layer k, DCR i is the computing resource amount of the ith data center, Cal y is the computing amount of the network layer y, DCR x is the computing resource amount of the xth data center, n is the total number of data centers, m is the total number of network layers in the large model to be trained, and i is the number of the current data center to be allocated.

[0020] Preferably, the pipeline slicing method is used to control the parallel training of the network layers divided by each data center, and load balancing is performed after the training of each batch of samples is completed, including:

[0021] At the beginning of the training of each batch of samples, the data center located at the starting position in the shortest path is controlled to divide the batch of samples into a plurality of micro-batches of samples;

[0022] In the forward propagation stage of the loss calculation of the large model training, after the data center completes the calculation of a micro-batch of samples, the calculation result is sent to the next adjacent data center on the path, so that the next data center performs the calculation of the micro-batch of samples;

[0023] After the forward propagation is completed, each data center records the cumulative waiting time of the forward arrival of the micro-batch of samples other than the first micro-batch of data;

[0024] In the backward propagation stage of the gradient update of the large model training, after the data center completes the calculation of a micro-batch of samples, the calculation result is sent to the adjacent previous data center on the path, so that the previous data center performs the calculation of the micro-batch of samples;

[0025] After the backward propagation is completed, the control data center records the accumulated waiting time of the backward propagation of the micro-batch data other than the first micro-batch data;

[0026] After the training of the current round of training samples is completed, the accumulated waiting time recorded by each data center during the forward propagation and the backward propagation is used to re-allocate tasks to each data center to achieve load balancing.

[0027] Further, the data amount of the micro-batch sample is as follows:

[0028]

[0029] In the above formula, b is the data amount of the micro-batch sample, DCE x is the computing resource amount of the xth data center, DCCal x is the computing amount of the network layer of the xth data center, n is the total number of data centers, and B is the data amount of the batch sample.

[0030] Further, the process of load balancing includes:

[0031] When the difference between the forward accumulated waiting time of a data center and the forward accumulated waiting time of the previous data center exceeds a given threshold, the last layer of computing tasks of the previous data center is allocated to the current data center;

[0032] When the difference between the backward accumulated waiting time of a data center and the backward accumulated waiting time of the next data center exceeds a given threshold, the first layer of computing tasks of the next data center is allocated to the current data center.

[0033] In a second aspect, a device for optimizing the communication efficiency of a computing power network for large model training is provided, and the device includes:

[0034] An analysis module is configured to calculate the shortest path from a given data center to all remaining data centers based on the bidirectional transmission delay between the data centers.

[0035] An allocation module is configured to allocate the network layers to be trained to each data center based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained.

[0036] A training module is configured to control each data center to perform parallel training on the network layer allocated to it in a pipeline slicing manner, and perform load balancing after the training of each batch of samples is completed until the training of the large model to be trained is completed.

[0037] Preferably, the analysis module is specifically configured to:

[0038] constructing a transmission delay adjacency matrix;

[0039] using the backtracking method to traverse paths constructed by data centers in different orders with the given data center as the starting point of the path, and recording the corresponding paths and total transmission delays;

[0040] comparing the total transmission delays of all paths, and taking the path with the minimum total transmission delay as the shortest path.

[0041] Preferably, the allocation module is specifically used for:

[0042] constructing a constraint condition based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained;

[0043] allocating network layers to be trained to each data center with the goal of meeting the constraint condition.

[0044] Further, the constraint condition is as follows:

[0045]

[0046] In the above formula, layers j is the number of network layers allocated to the jth data center, Cal k is the computing amount of network layer k, DCR i is the computing resource amount of the ith data center, Cal y is the computing amount of network layer y, DCR x is the computing resource amount of the xth data center, n is the total number of data centers, m is the total number of network layers in the large model to be trained, and i is the number of the current data center to be allocated.

[0047] Preferably, the training module is specifically used for:

[0048] controlling the data center at the starting position in the shortest path to divide the batch of samples into several micro-batches of samples at the beginning of each batch of sample training;

[0049] controlling the data center to send the calculation results to the next adjacent data center on the path after completing the calculation of a micro-batch of samples in the forward propagation stage of loss calculation in large model training, so that the next data center performs the calculation of the micro-batch of samples;

[0050] controlling each data center to record the cumulative waiting time of the forward arrival of micro-batches of data other than the first micro-batch of data after the forward propagation ends;

[0051] In the back propagation phase of the gradient update of the large model training, the data center is controlled to send the calculation result to the adjacent last data center on the path after completing the calculation of one micro-batch sample, so that the last data center performs the calculation of the micro-batch sample.

[0052] After the back propagation is completed, the data center records the accumulated waiting time of the micro-batch data other than the first micro-batch data back to the last data center;

[0053] After the training of the current round of training samples is completed, the accumulated waiting time recorded by each data center in the forward propagation and back propagation is used to redivide the tasks of each data center to achieve load balancing.

[0054] Further, the data amount of the micro-batch sample is as follows:

[0055]

[0056] In the above formula, b is the data amount of the micro-batch sample, DCR x is the calculation resource amount of the xth data center, DCCal x is the calculation amount of the network layer of the xth data center, n is the total number of data centers, and B is the data amount of the batch sample.

[0057] Further, the process of load balancing includes:

[0058] When the difference between the forward accumulated waiting time of a data center and the forward accumulated waiting time of the last data center exceeds a given threshold, the last layer calculation task of the last data center is divided to the current data center;

[0059] When the difference between the reverse accumulated waiting time of a data center and the reverse accumulated waiting time of the next data center exceeds a given threshold, the first layer calculation task of the next data center is divided to the current data center.

[0060] In a third aspect, a computer device is provided, including: one or more processors;

[0061] The processor is configured to store one or more programs;

[0062] When the one or more programs are executed by the one or more processors, the large model training oriented computing power network communication efficiency optimization method is implemented.

[0063] In a fourth aspect, a computer readable storage medium is provided, which has a computer program stored thereon, and the computer program is executed to implement the large model training oriented computing power network communication efficiency optimization method.

[0064] The one or more technical solutions of the application have at least one or more of the following beneficial effects.

[0065] The application provides a computing power network communication efficiency optimization method and device for large model training, comprising: calculating a shortest path from a given data center to traverse all remaining data centers based on the bidirectional transmission time delay between data centers; allocating network layers to be trained to each data center based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained; and controlling parallel training of the network layers divided by each data center in a pipeline slicing manner, and performing load balancing after the training of each batch of samples is completed until the training of the large model to be trained is completed. The technical solution provided by the application finds a path from a specified data center to connect all remaining data centers and has the shortest time delay, reduces the overall data transmission time delay, greatly reduces the idle waiting time of the computing device by using the pipeline parallel computing data transmission method, thereby improving the utilization rate of computing resources and accelerating the model training process, and dynamically adjusts the task computing amount between data centers to make each data center receive and transmit data while performing computation as much as possible, so that the communication time is covered by the computation time as much as possible, thereby improving the utilization rate of computing resources. Therefore, by searching for the shortest path, pipeline parallel computing and communication, and dynamic load balancing, the application can effectively optimize the communication efficiency of the computing power network during large model training, improve the computing / communication ratio, thereby improving the resource utilization rate of the computing power network and accelerating the model training process. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the main step flow diagram of the computing power network communication efficiency optimization method for large model training of the embodiment of the application;

[0067] Figure 2 is the structure diagram of the computing power network communication efficiency optimization system for large model training of the embodiment of the application;

[0068] Figure 3 is the shortest path selection process diagram based on transmission time delay sensing of the embodiment of the application;

[0069] Figure 4 is the task division process diagram of the embodiment of the application;

[0070] Figure 5 is the forward propagation and backward propagation pipeline working diagram of the embodiment of the application;

[0071] Figure 6 is the process diagram of the load balancing module adjusting the model layer of the embodiment of the application. DETAILED DESCRIPTION

[0072] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0073] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0074] As disclosed in the background, in recent years, with the continuous improvement and progress of deep learning (DL) technology, artificial intelligence has been widely applied in many fields, such as computer vision, natural language processing, autonomous driving, and speech recognition. However, traditional small models are increasingly unable to keep up with the times, and for this reason, a large number of researchers continue to expand the size of models and datasets, which has brought problems such as significant extension of model training time and insufficient memory space. Although hardware manufacturers are constantly improving the performance of computing chips, it is still not enough to directly solve these problems from the hardware level. In order to speed up the training of models and support the training of larger models, the academic and industrial communities usually use Distributed Deep Learning (DDL) technology to train large models, that is, multiple GPUs or even multiple GPU servers are used to train in parallel within a GPU cluster / data center. Data centers have to continuously expand their scale to accommodate the growing number of model parameters. For the training of super-large models, such as GPT-3 with 175 billion parameters and PaLM with 540 billion parameters, a data center may need thousands or even tens of thousands of GPUs to complete the training within an acceptable time.

[0075] It can be predicted that the size of models will continue to grow, and it will be difficult for a single data center to support such training. Therefore, the computing power network is proposed. As a new type of network architecture, the computing power network not only faces the major needs of the current development of China's digital economy, but also aims to build a self-controllable and functionally leading computing network integration system, realize intelligent, efficient, and flexible computing resource integration and scheduling, meet the differentiated computing power service needs of industries, and provide support for the development of computing power and intelligent industries. Although the computing power network is based on the traditional computer network, how to realize efficient communication between nodes to meet the training needs of artificial intelligence large models is still a key problem.

[0076] In the computing power network, artificial intelligence large models are usually trained in parallel by multiple data centers in a model-parallel manner. The huge amount of original sample data and a large amount of intermediate data generated during the training process will generate huge communication overhead when transmitted between data centers. When the communication time that does not overlap with the computing time accounts for a high proportion in the total time of training, it will seriously affect the efficiency of model training. If a traditional communication method is used to transmit data between data centers, problems such as too large transmission delay and a large number of GPUs idle for waiting for data may occur. If a new efficient communication protocol is designed for this purpose, it is not only difficult, but also difficult to implement in the short term. The method of upgrading the transmission medium to obtain greater bandwidth depends more on a major breakthrough in materials science. Therefore, there is an urgent need for a feasible method to reduce communication overhead by optimizing data transmission methods on the basis of traditional transmission media and communication protocols.

[0077] To improve the above problems, the application provides a computing power network communication efficiency optimization method and device for large model training, comprising: based on the bidirectional transmission delay between data centers, the shortest path from a given data center to all remaining data centers is calculated; based on the computing resource amount of each data center on the shortest path and the computing amount of each network layer in the large model to be trained, each data center is assigned the network layer to be trained; the pipeline slicing method is used to control each data center to perform parallel training on the network layer divided to it, and load balancing is performed after each batch of sample training is completed until the large model to be trained is trained. The technical scheme provided by the application reduces the overall data transmission delay by finding a path from a specified data center that connects all remaining data centers and has the shortest delay. By using the pipeline parallel computing data transmission method, the idle waiting time of the computing device can be greatly reduced, thereby improving the utilization of computing resources and accelerating the model training process. By dynamically adjusting the task computing amount between data centers, each data center can receive and transmit data while performing calculations as much as possible, so that the communication time is covered by the computing time as much as possible, thereby improving the utilization of computing resources. Therefore, by searching for the shortest path, pipeline parallel computing and communication, and dynamic load balancing, the application can effectively optimize the communication efficiency of the computing power network during large model training, improve the computing / communication ratio, and thus improve the resource utilization of the computing power network and accelerate the model training process.

[0078] The above scheme will be described in detail below.

[0079] Embodiment 1

[0080] Refer to the accompanying Figure 1 , Figure 1 is the main step flowchart of the large model training-oriented computing power network communication efficiency optimization method of an embodiment of the application. As shown in Figure 1As shown, the algorithm power network communication efficiency optimization method for large model training in the embodiment of the application mainly includes the following steps:

[0081] Step S101: Based on the bidirectional transmission delay between data centers, the shortest path from a given data center to traverse all remaining data centers is calculated.

[0082] In this embodiment, although the selected data centers are usually network reachable, not all data centers are directly reachable between each other, and some data centers may need to rely on other data centers for communication. Bidirectional communication between two data centers may use different physical links, so it is necessary to perceive the bidirectional transmission delay between the two data centers. The bidirectional transmission delay can be obtained by adding two one-way transmission delays, and the one-way transmission delay between two data centers can be obtained by sending an ICMP message test to the target host.

[0083] By abstracting each data center as a node in an undirected graph and abstracting the bidirectional transmission delay between data centers as the path length, and using the delay data obtained by the transmission delay perception module to construct an adjacency matrix to format the path length between nodes, the adjacency matrix is used as input, and the backtracking method is used to find a shortest path from a specified node and traverse all remaining nodes.

[0084] Step S102: Based on the computing resources of each data center on the shortest path and the computing amount of each network layer in the large model to be trained, each data center is allocated the network layer to be trained.

[0085] In this embodiment, by parsing the source code defined by the large model training, the hierarchical structure of the large model is obtained, and each layer source code is further parsed to evaluate the computing amount and memory requirement of each layer for a single sample through its parameters and operation logic.

[0086] In order to fully exert the efficiency of pipeline slicing calculation and transmission, data centers with more computing resources need to perform more model layer calculations, while data centers with less computing resources need to perform less model layer calculations. The initial division result may not be accurate, but it does not affect the implementation of the method, and dynamic adjustment will still be performed in the subsequent execution process.

[0087] Step S103: Adopting a pipeline slicing method to control each data center to perform parallel training on the network layer divided to it, and performing load balancing after the training of each batch of samples is completed, until the training of the large model to be trained is completed.

[0088] In this embodiment, in order to improve the efficiency of pipeline fragment calculation and transmission, the batch size needs to be divided into smaller micro-batch size, so that multiple data centers can work simultaneously to improve the efficiency of the pipeline. Due to the dynamic changes of the working nodes of each data center, the load of each data center is unbalanced. The waiting time recorded by each data center is used to determine whether the imbalance occurs. If the imbalance occurs, the model layer calculation task is transferred between adjacent data centers to rebalance the load of each data center.

[0089] In one embodiment, based on the bidirectional transmission delay between data centers, the shortest path from a given data center to all remaining data centers is calculated, including:

[0090] Construct a transmission delay adjacency matrix;

[0091] Take a given data center as the starting point of the path, use backtracking to traverse the path constructed by data centers in different orders, and record the corresponding path and total transmission delay;

[0092] Compare the total transmission delay of all paths, and take the path with the smallest total transmission delay as the shortest path.

[0093] For example: by abstracting each data center as a node in an undirected graph and abstracting the bidirectional transmission delay between data centers as the path length, and using the delay data obtained by the transmission delay perception module to construct an adjacency matrix to format the path length between nodes. For n data centers, the adjacency matrix should be an n-order square matrix, where the value of the ith row and jth column of the adjacency matrix c ij represents the bidirectional transmission delay between the ith data center and the jth data center. ij Then take the adjacency matrix as input and use backtracking to find a shortest path from a specified node and traverse all remaining nodes.

[0094] In this embodiment, based on the computing resources of each data center on the shortest path and the computing amount of each network layer in the large model to be trained, each data center is allocated a network layer to be trained, including:

[0095] Based on the computing resources of each data center on the shortest path and the computing amount of each network layer in the large model to be trained, a constraint condition is constructed;

[0096] Each data center is allocated a network layer to be trained to meet the constraint condition.

[0097] In one embodiment, the constraint condition is as follows:

[0098]

[0099] In the above formula, layers j is the number of network layers allocated to the jth data center, Cal k is the calculation amount of network layer k, DCR i is the calculation resource amount of the ith data center, Cal y is the calculation amount of network layer y, DCR x is the calculation resource amount of the xth data center, n is the total number of data centers, m is the total number of network layers in the large model to be trained, and i is the number of the current data center to be allocated.

[0100] Since the calculation amount counted by the model layer as the minimum division unit is not continuously changed, the number of model layers divided according to the proportion of the data center computing resource and the model layer calculation amount is not an integer. In order to prevent the load amount of the last data center from being too large due to the downward rounding, the upward rounding is selected in this embodiment.

[0101] In this embodiment, the pipeline fragmentation manner is used to control the parallel training of the network layers allocated to each data center, and load balancing is performed after the training of each batch of samples is completed, which includes:

[0102] At the beginning of the training of each batch of samples, the data center located at the starting position in the shortest path is controlled to divide the batch of samples into a plurality of micro-batches of samples;

[0103] In the forward propagation stage of the loss calculation of the large model training, after the data center completes the calculation of a micro-batch of samples, the calculation result is sent to the next adjacent data center on the path, so that the next data center performs the calculation of the micro-batch of samples;

[0104] After the forward propagation is completed, the data centers are controlled to record the cumulative waiting time of the forward arrival of the micro-batch data other than the first micro-batch data;

[0105] In the backward propagation stage of the gradient update of the large model training, after the data center completes the calculation of a micro-batch of samples, the calculation result is sent to the adjacent previous data center on the path, so that the previous data center performs the calculation of the micro-batch of samples;

[0106] After the backward propagation is completed, the data centers are controlled to record the cumulative waiting time of the backward arrival of the micro-batch data other than the first micro-batch data;

[0107] After the training of the current round of training samples is completed, the cumulative waiting time recorded by each data center in the forward propagation and backward propagation processes is used to re-allocate tasks to each data center to achieve load balancing.

[0108] In order to perform pipeline fragment calculation and transmission efficiently, the batch size needs to be divided into smaller micro-batch size so that multiple data centers can work simultaneously to improve the efficiency of the pipeline; due to the dynamic changes of the working nodes of each data center, the load of each data center is unbalanced, and the waiting time recorded by each data center is used to determine whether the imbalance occurs, and if the imbalance occurs, the model layer calculation task is transferred between adjacent data centers to rebalance the load of each data center.

[0109] In one embodiment, the data amount of the micro-batch sample is as follows:

[0110]

[0111] In the above formula, b is the data amount of the micro-batch sample, DCR x is the computing resource amount of the xth data center, DCCal x is the computing amount of the network layer of the xth data center, n is the total number of data centers, and B is the data amount of the batch sample.

[0112] In one embodiment, the load balancing process includes:

[0113] When the difference between the forward cumulative waiting time of a data center and the forward cumulative waiting time of the previous data center exceeds a given threshold, the last layer of calculation task of the previous data center is divided to the current data center;

[0114] When the difference between the reverse cumulative waiting time of a data center and the reverse cumulative waiting time of the next data center exceeds a given threshold, the first layer of calculation task of the next data center is divided to the current data center.

[0115] In order to implement the above method, the present application provides a computing power network communication efficiency optimization system for large model training, which is used to execute the computing power network communication efficiency optimization method described above, and the computing power network communication efficiency optimization system for large model training includes:

[0116] A transmission delay awareness module is used to perceive the bidirectional transmission delay between data centers;

[0117] A path selection module is used to establish an adjacency matrix using the bidirectional transmission delay data between data centers obtained by the transmission delay awareness module, and to search for the transmission path with the shortest delay using the backtracking method;

[0118] A model analysis module is used to analyze the network layer structure of the large model to be trained and to evaluate the computing amount of each layer;

[0119] A task division module is used to divide the training task of the artificial intelligence large model according to its model layer and to assign it to the corresponding data center.

[0120] a computing module for performing model training tasks, composed of individual data centers and a communication system;

[0121] a load balancing module for dynamically adjusting the amount of tasks of each data center when the data center load is unbalanced.

[0122] Please refer to Figure 2 ; Figure 2 The architecture of the computing power network communication efficiency optimization system proposed in the application is given. It contains a transmission delay awareness module, a path selection module, a task division module, a computing module and a load balancing module. The horizontal arrows between data centers in the computing module represent forward propagation and back propagation, and the shaded part represents the model layer migration that the load balancing module may perform.

[0123] The transmission delay awareness module, the path selection module, the model analysis module, the task division module, the computing module and the load balancing module are introduced below.

[0124] (1) Transmission delay awareness module

[0125] Since the training of a neural network model involves forward propagation and back propagation, bidirectional communication between data centers is also required when training a large model using a computing power network. However, the physical links used for bidirectional communication between two data centers may not be the same, resulting in different bidirectional transmission delays. Therefore, the transmission delay awareness module in this method needs to be aware of the bidirectional transmission delay between data centers. This module is only a necessary preliminary work for this method and system, so the specific delay awareness technology is not the focus of innovation in this method. There are many ways to implement transmission delay awareness, such as testing transmission delay through ICMP packets. Different implementation methods do not affect the subsequent implementation of this method.

[0126] (2) Path selection module

[0127] The path selection module uses the bidirectional delay data obtained by the transmission delay awareness module as the path length, and establishes an adjacency matrix to describe the paths between the selected data centers two by two, and uses a path search algorithm to calculate the shortest path from a given data center to all remaining data centers (without specifying the end data center).

[0128] (3) Model analysis module

[0129] The model analysis module first obtains the defined network layer structure of the large model to be trained through the source code of the large model, and then analyzes the calculation process and parameters of each layer to evaluate the computation and memory requirements of each sample of each layer.

[0130] (4) Task division module

[0131] The task division module in the method is used for dividing the training task of the artificial intelligence large model into multiple sub-tasks, i.e., the calculation amount and the memory requirement of each layer evaluated by the model analysis module are proportionally divided to each data center according to the available resources of the data center for arranging the transmission path.

[0132] (5) The calculation module

[0133] The calculation module is the core of the large model training system, and all training tasks are completed by the calculation module. The calculation module includes a plurality of selected data centers, and each data center performs forward propagation and backward propagation in a pipeline manner in a good arrangement order. Specifically, at the beginning of each batch of sample training, the data center at the starting position of the path reasonably divides the batch size into a plurality of smaller micro-batches. In the forward propagation stage, each data center only performs the calculation steps of the corresponding model layer for one micro-batch at a time. After completing the calculation of one micro-batch, the calculation result is immediately sent to the next adjacent data center on the path, while the current data center will also perform the calculation of the next micro-batch. In addition, each data center will record the cumulative waiting time of the forward arrival of micro-batch data other than the first micro-batch data. In the backward propagation stage, each data center still only performs the calculation steps of the corresponding model layer for one micro-batch at a time. After completing the calculation of one micro-batch, the calculation result is immediately sent to the previous adjacent data center on the path, while the current data center will also perform the calculation of the next micro-batch. In addition, each data center will record the cumulative waiting time of the backward arrival of micro-batch data other than the first micro-batch data.

[0134] (6) The load balancing module

[0135] The load balancing module determines whether the task quantity of each data center is unbalanced in the large model training process through the waiting time recorded by each data center, and dynamically adjusts the task quantity of each data center to make the pipeline work more efficiently if unbalance occurs. Specifically, if the difference between the forward cumulative waiting time of a data center and the forward cumulative waiting time of the previous data center exceeds a certain threshold, the last layer calculation task of the previous data center is divided to the current data center; if the difference between the backward cumulative waiting time of a data center and the backward cumulative waiting time of the next data center exceeds a certain threshold, the first layer calculation task of the next data center is divided to the current data center.

[0136] Please refer to Figure 3, in order to reduce the communication overhead caused by transmission delay as much as possible, the present application proposes a method for reducing transmission delay by constructing the shortest path. Each data center is abstracted as a node in an undirected graph, and the bidirectional transmission delay between data centers is abstracted as the path length, and an adjacency matrix is constructed to format the path length between nodes. For n data centers, the adjacency matrix should be an n-order square matrix, where the value of the i-th row and the j-th column in the adjacency matrix c ij represents the bidirectional transmission delay between the i-th data center and the j-th data center c ij . The shortest path algorithm uses the adjacency matrix to find the shortest path from a given node to all other nodes to reduce the total transmission delay.

[0137] Please refer to Figure 4 , each data center cooperates to train the large model in a model parallel manner, so the original model is divided according to its network layer structure, and the calculation amount and required memory size of each layer are evaluated, and then the network layer is divided in proportion to the calculation resources of each data center.

[0138] Please refer to Figure 5 , the present application proposes a pipeline working mode for computing and communicating between selected data centers to improve the calculation / communication ratio of the computing power network. The zebra pattern block represents the forward propagation process of the sliced data, and the grid pattern block represents the backward propagation process of the sliced data. The data center at the starting position of the path first slices the original batch size of sample data, and sends the sliced data to the next data center immediately after each calculation; the last data center on the path calculates the gradient in the order of receiving all the sliced data, and sends the data to the previous data center immediately after each calculation. There is a transmission delay between data centers, so after the data is sent by the previous data center, the next data center needs to wait for a certain delay before receiving the corresponding data and performing calculation.

[0139] Please refer to Figure 6 , the present application proposes a load balancing module to ensure the execution efficiency of the pipeline. A load imbalance situation is shown in the figure: the slice execution time of data center 2 is obviously larger than that of data center 1, in the forward propagation, the transmission time of other sliced data except the first sliced data can be covered by the sliced data execution time of data center 2, but in the backward propagation, since the calculation task amount of data center 2 is larger, data center 1 may have executed the first sliced data before data center 2 executes the second sliced data, resulting in idle waiting time for the second sliced data; data center n also has idle waiting time in the forward propagation. The load balancing module adjusts the task distribution between data centers according to the idle waiting time recorded by each data center in the forward propagation and the backward propagation.

[0140] The implementation process of the present application is applicable to different large model training tasks, and is not concerned with which computing devices are used by each data center. Each data center can be adapted to the implementation process of the present application by abstracting the computing power resources of the computing devices to shield their heterogeneity. The computing power network includes at least two data centers, and each data center has at least one computing device (CPU, GPU, FPGA, etc.), and the total available computing power resources of each data center must be sufficient to support the training of the target large model.

[0141] Embodiment 2

[0142] Based on the same inventive concept, the present application also provides a computing power network communication efficiency optimization device for large model training, which comprises:

[0143] An analysis module is configured to calculate the shortest path from a given data center to all remaining data centers based on the bidirectional transmission delay between the data centers.

[0144] An allocation module is configured to allocate the network layers to be trained to each data center based on the computing resources of each data center on the shortest path and the computing amount of each network layer in the large model to be trained.

[0145] A training module is configured to control the parallel training of each data center on the network layer divided by it in a pipeline slicing manner, and perform load balancing after the training of each batch of samples is completed until the training of the large model to be trained is completed.

[0146] Preferably, the analysis module is specifically configured to:

[0147] Construct a transmission delay adjacency matrix;

[0148] Use the backtracking method to traverse the paths constructed by the data centers in different orders with the given data center as the starting point of the path, and record the corresponding path and total transmission delay;

[0149] Compare the total transmission delays of all paths, and take the path with the smallest total transmission delay as the shortest path.

[0150] Preferably, the allocation module is specifically configured to:

[0151] Construct a constraint condition based on the computing resources of each data center on the shortest path and the computing amount of each network layer in the large model to be trained;

[0152] Allocate the network layers to be trained to each data center to meet the constraint condition.

[0153] Further, the constraint condition is as follows:

[0154]

[0155] In the above formula, layers j is the number of network layers allocated to the jth data center, Cal k is the calculation amount of network layer k, DCR i is the calculation resource amount of the ith data center, Cal y is the calculation amount of network layer y, DCR x is the calculation resource amount of the xth data center, n is the total number of data centers, m is the total number of network layers in the large model to be trained, and i is the number of the current data center to be allocated.

[0156] Preferably, the training module is specifically used for:

[0157] At the beginning of training of each batch of samples, the data center located at the starting position in the shortest path divides the batch of samples into a plurality of micro-batches of samples;

[0158] In the forward propagation stage of loss calculation of large model training, after a data center completes the calculation of a micro-batch of samples, the data center sends the calculation result to the next adjacent data center on the path, so that the next adjacent data center performs the calculation of the micro-batch of samples;

[0159] After the forward propagation ends, the data centers record the cumulative waiting time of the forward arrival of micro-batches of data other than the first micro-batch of data;

[0160] In the backward propagation stage of gradient update of large model training, after a data center completes the calculation of a micro-batch of samples, the data center sends the calculation result to the previous adjacent data center on the path, so that the previous adjacent data center performs the calculation of the micro-batch of samples;

[0161] After the backward propagation ends, the data centers record the cumulative waiting time of the backward arrival of micro-batches of data other than the first micro-batch of data;

[0162] After completing the training of the current round of training samples, the cumulative waiting times recorded by the data centers in the forward propagation and backward propagation processes are used to re-allocate tasks to the data centers to achieve load balancing.

[0163] Further, the data amount of the micro-batch of samples is as follows:

[0164]

[0165] In the above formula, b is the data amount of the micro-batch of samples, DCR x is the calculation resource amount of the xth data center, DCCal xThe calculation amount of the network layer of the xth data center, n is the total number of data centers, and B is the data amount of the batch sample.

[0166] Further, the load balancing process includes:

[0167] When the difference between the forward cumulative waiting time of a data center and the forward cumulative waiting time of the previous data center exceeds a given threshold, the last layer of computing tasks of the previous data center is divided to the current data center.

[0168] When the difference between the reverse cumulative waiting time of a data center and the reverse cumulative waiting time of the next data center exceeds a given threshold, the first layer of computing tasks of the next data center is divided to the current data center.

[0169] Embodiment 3

[0170] Based on the same inventive concept, the application further provides a computer device, which comprises a processor and a memory. The memory is used to store a computer program, and the computer program comprises program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor is the computing core and control core of the terminal, and is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function, so as to implement the steps of the above-mentioned power network communication efficiency optimization method for large model training.

[0171] Embodiment 4

[0172] Based on the same inventive concept, the present application also provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the computer device, and of course can also include the expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the steps of the above-mentioned power network communication efficiency optimization method for large model training in the embodiments.

[0173] Those skilled in the art will appreciate that embodiments of the present application can be supplied as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0175] These computer program instructions can also be stored in a computer readable memory capable of directing the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocksFigure 1 the function specified in the one or more blocks.

[0176] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processing, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0177] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the scope of protection of the claims of the present application.

Claims

1. A method for optimizing the communication efficiency of computing power networks for large model training, characterized in that, The method includes: Based on the bidirectional transmission latency between data centers, the shortest path is calculated to traverse all remaining data centers from a given data center. Based on the amount of computing resources in each data center along the shortest path and the amount of computing resources in each network layer of the large model to be trained, the network layers to be trained are allocated to each data center. A pipeline-based sharding approach is adopted to control each data center to perform parallel training on its assigned network layer, and load balancing is performed after the training of each batch of samples is completed, until the training of the large model to be trained is completed. The method for calculating the shortest path from a given data center to all remaining data centers, based on the bidirectional transmission latency between data centers, includes: Construct a transmission delay adjacency matrix; Starting from a given data center, use backtracking to traverse paths constructed from data centers in different orders, and record the corresponding paths and total transmission delays. Compare the total transmission delay of all paths and select the path with the minimum total transmission delay as the shortest path; The allocation of network layers to be trained to each data center based on the computational resources of each data center along the shortest path and the computational resources of each network layer in the large model to be trained includes: Constraints are constructed based on the amount of computing resources in each data center along the shortest path and the amount of computing resources in each network layer of the large model to be trained. To satisfy the aforementioned constraints, network layers to be trained are allocated to each data center. The constraints are as follows: In the above formula, layers j For the number of network layers allocated to the j-th data center, Cal k DCR is the computational cost of network layer k. i Let Cal be the amount of computing resources for the i-th data center. y The computational cost of network layer y, DCR x Let be the computing resources of the x-th data center, n be the total number of data centers, m be the total number of network layers in the large model to be trained, and i be the number of the data center to be allocated. The method of using a pipelined sharding approach to control the parallel training of the network layers allocated to each data center, and performing load balancing after the training of each batch of samples, includes: At the start of training for each batch of samples, the data center located at the starting position in the shortest path is controlled to divide the batch of samples into several micro-batches. During the forward propagation phase of loss calculation in large model training, after the control data center completes the calculation of a micro-batch of samples, it sends the calculation result to the next adjacent data center on the path so that the next data center can perform the calculation of the micro-batch of samples. After the forward propagation is completed, control each data center to record the cumulative waiting time for the arrival of other micro-batch data except for the first micro-batch data; During the backpropagation phase of gradient updates in large model training, after the control data center completes the calculation of a micro-batch of samples, it sends the calculation result to the adjacent data center on the path so that the previous data center can perform the calculation of the micro-batch of samples. After the back propagation is completed, the control data center records the cumulative waiting time for the back-arrival of all micro-batch data except for the first micro-batch data; After completing the training of this round of training samples, the tasks for each data center are re-divided according to the cumulative waiting time recorded by each data center during the forward and backward propagation processes to achieve load balancing.

2. The method as described in claim 1, characterized in that, The data volume of the micro-batch samples is as follows: In the above formula, b is the data size of the micro-batch sample, and DCR x Let x be the computing resources of the x-th data center, DCCal x Let B be the computational cost of the network layer for the x-th data center, n be the total number of data centers, and B be the data volume of the batch sample.

3. The method as described in claim 2, characterized in that, The load balancing process includes: When the difference between the forward cumulative latency of a data center and the forward cumulative latency of the previous data center exceeds a given threshold, the last layer of computing tasks in the previous data center will be assigned to the current data center. When the difference between the reverse cumulative wait time of one data center and the reverse cumulative wait time of the next data center exceeds a given threshold, the first layer of computing tasks of the next data center is assigned to the current data center.

4. A computing power network communication efficiency optimization device for large model training, characterized in that, The device includes: The analysis module is used to calculate the shortest path from a given data center to all remaining data centers based on the bidirectional transmission latency between data centers. The allocation module is used to allocate the network layers to be trained to each data center based on the amount of computing resources of each data center on the shortest path and the amount of computing resources of each network layer in the large model to be trained. The training module is used to control each data center to perform parallel training of its assigned network layer using a pipelined sharding approach, and to perform load balancing after the training of each batch of samples is completed, until the training of the large model to be trained is completed. The analysis module is specifically used for: Construct a transmission delay adjacency matrix; Starting from a given data center, use backtracking to traverse paths constructed from data centers in different orders, and record the corresponding paths and total transmission delays. Compare the total transmission delay of all paths and select the path with the minimum total transmission delay as the shortest path; The allocation module is specifically used for: Constraints are constructed based on the amount of computing resources in each data center along the shortest path and the amount of computing resources in each network layer of the large model to be trained. To satisfy the aforementioned constraints, network layers to be trained are allocated to each data center. The constraints are as follows: In the above formula, layers j For the number of network layers allocated to the j-th data center, Cal k DCR is the computational cost of network layer k. i Let Cal be the amount of computing resources for the i-th data center. y The computational cost of network layer y, DCR x Let be the computing resources of the x-th data center, n be the total number of data centers, m be the total number of network layers in the large model to be trained, and i be the number of the data center to be allocated. The training module is specifically used for: At the start of training for each batch of samples, the data center located at the starting position in the shortest path is controlled to divide the batch of samples into several micro-batches. During the forward propagation phase of loss calculation in large model training, after the control data center completes the calculation of a micro-batch of samples, it sends the calculation result to the next adjacent data center on the path so that the next data center can perform the calculation of the micro-batch of samples. After the forward propagation is completed, control each data center to record the cumulative waiting time for the arrival of other micro-batch data except for the first micro-batch data; During the backpropagation phase of gradient updates in large model training, after the control data center completes the calculation of a micro-batch of samples, it sends the calculation result to the adjacent data center on the path so that the previous data center can perform the calculation of the micro-batch of samples. After the back propagation is completed, the control data center records the cumulative waiting time for the back-arrival of all micro-batch data except for the first micro-batch data; After completing the training of this round of training samples, the tasks for each data center are re-divided according to the cumulative waiting time recorded by each data center during the forward and backward propagation processes to achieve load balancing.

5. The apparatus as described in claim 4, characterized in that, The data volume of the micro-batch samples is as follows: In the above formula, b is the data size of the micro-batch sample, and DCR x Let x be the computing resources of the x-th data center, DCCal x Let B be the computational cost of the network layer for the x-th data center, n be the total number of data centers, and B be the data volume of the batch sample.

6. The apparatus as claimed in claim 5, characterized in that, The load balancing process includes: When the difference between the forward cumulative latency of a data center and the forward cumulative latency of the previous data center exceeds a given threshold, the last layer of computing tasks in the previous data center will be assigned to the current data center. When the difference between the reverse cumulative wait time of one data center and the reverse cumulative wait time of the next data center exceeds a given threshold, the first layer of computing tasks of the next data center is assigned to the current data center.

7. A computer device, characterized in that, include: One or more processors; The processor is used to execute one or more programs; When the one or more programs are executed by the one or more processors, the method for optimizing computing power network communication efficiency for large model training as described in any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that, It contains a computer program, which, when executed, implements the method for optimizing the computing power network communication efficiency for large model training as described in any one of claims 1 to 3.

Images