A large-scale distributed training method and related equipment for cloud environments

Abstract

In response to the scenario requirements of using the computing power of cross-regional computing centers for deep model training, this paper designs a deep model training method for cloud environments. It adopts strategies such as splitting model parameters according to network layers to different computing centers, using compressed communication between cross-domain computing nodes, and using hybrid parallelism within and between computing centers to achieve the collaborative completion of deep model training using cross-domain computing power. This invention can make large model training no longer restricted by the computing power resources of a single computing center, and can efficiently utilize the computing power of cross-domain computing centers, and use cross-domain computing power to collaboratively complete large-scale model training tasks in a low-bandwidth environment.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a large-scale distributed training method, system, terminal and computer-readable storage medium for a cloud environment. Background Art

[0002] Currently, the parameter scale of deep learning models is developing towards an increasingly larger trend, and the computing power resources required for model training are also increasing. For the training of large-scale models, a single computing center is often unable to provide all the computing power required for training. In this case, if the computing power resources of cross-regional computing centers can be used to complete the model training task, then the problem of insufficient computing power resources for large model training can be effectively solved.

[0003] Most existing model training methods that utilize the computing power of cross-domain computing centers use a data parallel mode. Different computing centers use different data subsets to train model parameters. During the training process, a central parameter service node or multiple layered parameter service nodes aggregate and fuse the model parameters on each training node, and then send the fused model parameters to each training node for further training. This type of method requires training all model parameters on each training node, and full parameter synchronization during training. The amount of data transmission between the central node and the training nodes is large. In a low-bandwidth network environment, model parameters can only be synchronized at a low frequency. If the parameter synchronization frequency is too low, the convergence effect of the model training will be affected, and it may even make the model difficult to converge. If the parameter synchronization frequency is too high, the data transmission time during training will increase, and the effect of accelerating training by utilizing cross-domain computing power will not be achieved. In addition, because each training node needs to train all model parameters, this type of method is still limited by computing power. Each node participating in the training must provide sufficient computing power to support model training.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a large-scale distributed training method, system, terminal and computer-readable storage medium for cloud environments, aiming to solve the problems in the prior art that users' model training tasks are limited by the computing power of a single computing center, there is insufficient computing power resources for large-scale model training, and there is a lack of high-bandwidth network connections between cross-domain computing centers.

[0006] To achieve the above objectives, the present invention provides a large-scale distributed training method for cloud environments, which includes the following steps:

[0007] Upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task;

[0008] According to the resources required for the task and the resources in the several training nodes, the task is split into subtasks according to the hierarchy of the model, the subtasks are correspondingly assigned to the several training nodes, and a training process of the model is set;

[0009] According to the training process, the data is split into several batches of data, each batch of data is split into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, each mini-batch is sent to the first training node, and the data label corresponding to each mini-batch is sent to the last training node;

[0010] Perform forward calculation on each mini-batch to obtain the calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and fuse the reverse gradient corresponding to each mini-batch to obtain the target gradient. Update the model parameters on each training node according to the target gradient until the data training of each batch is completed;

[0011] The evaluation data is split into several batches of data, each batch of data is split into m mini-batches, each mini-batch is input into the training node after the model parameters are updated for calculation, and the final calculation result is obtained. The final calculation result is evaluated according to the data label corresponding to each mini-batch to obtain the evaluation result, until the evaluation of each batch of data is completed, and the overall evaluation result is obtained based on all the evaluation results. If the overall evaluation result meets the end condition of model training or the number of iterations, the training task is terminated.

[0012] Optionally, in the large-model distributed training method for cloud environments, the resources required for the task and the resources in the several training nodes include: computing power, video memory, internal memory and network bandwidth.

[0013] Optionally, the large-model distributed training method for a cloud environment, wherein the task is split according to the resources required for the task and the resources in the plurality of training nodes according to the hierarchy of the model to obtain subtasks, specifically includes:

[0014] Splitting the training task of the model parameters according to the model hierarchy of the model to obtain subtasks according to the resources required by the training task and the resources in the plurality of training nodes;

[0015] The model is split at the location where the scale of intermediate data transferred between layers is the smallest.

[0016] Optionally, in the large-scale distributed training method for a cloud environment, the process of setting up the training of the model specifically includes:

[0017] Optimizing and setting the model training process based on the resource conditions of the several training nodes and the scale of the model parameters, wherein the optimization includes parallel optimization and communication optimization, and the setting includes setting the end condition and the number of iteration rounds;

[0018] Among them, the parallel optimization refers to selecting a parallel training strategy based on the computing power and network bandwidth of the task and the several training nodes; the communication optimization refers to using a compression algorithm that meets the model accuracy loss requirements for the transmission data between the several training nodes.

[0019] Optionally, the large-model distributed training method for the cloud environment, wherein the forward calculation of each mini-batch to obtain the calculation result specifically includes:

[0020] Performing forward calculation on each mini-batch on the first training node, compressing a first intermediate result output by the forward calculation using the compression algorithm and sending the result to the second training node, and decompressing the first intermediate result after receiving the first intermediate result at the second training node;

[0021] The decompressed first intermediate result is forward-calculated on the second training node, and the calculated second intermediate result is compressed using the compression algorithm and sent to the third training node. The calculation is performed in sequence according to the model level until the calculation result is obtained at the last training node.

[0022] Optionally, the large-scale distributed training method for cloud environments, wherein the reverse calculation is performed based on the calculation results and the data labels corresponding to the data to obtain the reverse gradient, and each of the reverse gradients is fused to obtain the target gradient, specifically including:

[0023] Calculate the loss based on the calculation result at the last training node and the data label corresponding to the data, calculate the reverse gradient using the back propagation algorithm based on the calculated loss, compress the reverse gradient calculation result of the last training node using the compression algorithm and send it to the penultimate training node, the penultimate training node continues to perform reverse calculation and compresses and sends the calculation result, and calculates in sequence until the first training node obtains the reverse gradient of each mini-batch;

[0024] The reverse gradient of each mini-batch is fused to obtain the target gradient.

[0025] Optionally, the large-scale distributed training method for cloud environments further includes:

[0026] The process of fusing the reverse gradient and the process of updating the model parameters on each training node are both performed on the training node corresponding to the subtask.

[0027] Optionally, the large-model distributed training method for an inter-cloud environment, wherein the step of inputting each mini-batch into a training node after updating the model parameters for calculation to obtain a final calculation result, specifically includes:

[0028] Complete the forward calculation for the first training node of each mini-batch after updating the model parameters;

[0029] The intermediate result output by the forward calculation is compressed using the compression algorithm and sent to the second training node. The calculation is performed in sequence according to the model level until the last training node after the model parameters are updated to obtain the final calculation result.

[0030] Optionally, the large-scale distributed training method for cloud environments, wherein the step of evaluating the final calculation result according to the data labels corresponding to each mini-batch to obtain the evaluation result, specifically includes:

[0031] At the last training node after the model parameters are updated, the final calculation result is evaluated according to the data label corresponding to each mini-batch, and the evaluation result of the model on each mini-batch is calculated.

[0032] Optionally, the large-scale distributed training method for a cloud environment further includes:

[0033] If the overall evaluation result does not meet the end conditions or number of iterations of model training, the model will continue to be trained and evaluated.

[0034] In addition, to achieve the above-mentioned purpose, the present invention further provides a large-scale model distributed training system for a cloud environment, wherein the large-scale model distributed training system for a cloud environment comprises:

[0035] Information preparation module: upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task;

[0036] Task splitting module: splits the task according to the resources required by the task and the resources in the training nodes according to the hierarchy of the model to obtain subtasks, assigns the subtasks to the training nodes, and sets the training process of the model;

[0037] Data splitting module: Split the data into several batches according to the set training process, split each batch of data into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, send each mini-batch to the first training node, and send the data label corresponding to each mini-batch to the last training node;

[0038] Model training module: Perform forward calculation on each mini-batch to obtain the calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and fuse each of the reverse gradients to obtain the target gradient. Based on the target gradient, the model parameters on each training node are updated until the data training of each batch is completed;

[0039] Model evaluation module: Split the evaluation data into several batches of data, split each batch of data into m mini-batches, input each mini-batch into the training node after updating the model parameters for calculation, and obtain the final calculation result. Evaluate the final calculation result according to the data label corresponding to each mini-batch to obtain the evaluation result until the evaluation of each batch of data is completed. Get the overall evaluation result based on all the evaluation results. If the overall evaluation result meets the end condition or number of iterations of model training, the training task is terminated.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a large-model distributed training program for the cloud environment stored on the memory and runnable on the processor. When the large-model distributed training program for the cloud environment is executed by the processor, the steps of the large-model distributed training method for the cloud environment as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a large-model distributed training program for an inter-cloud environment, and when the large-model distributed training program for an inter-cloud environment is executed by a processor, the steps of the large-model distributed training method for an inter-cloud environment as described above are implemented.

[0042] In response to the scenario requirements of using the computing power of cross-regional computing centers for deep model training, this paper designs a deep model training method for cloud environments. It adopts strategies such as splitting model parameters according to network layers to different computing centers, using compressed communication between cross-domain computing nodes, and using hybrid parallelism within and between computing centers to achieve the collaborative completion of deep model training using cross-domain computing power. This invention can make large model training no longer restricted by the computing power resources of a single computing center, and can efficiently utilize the computing power of cross-domain computing centers, and use cross-domain computing power to collaboratively complete large-scale model training tasks in a low-bandwidth environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a preferred embodiment of the large-scale distributed training method for cloud environments of the present invention;

[0044] Figure 2 This is a schematic diagram of a preferred embodiment of the large-scale distributed training method for cloud environments of the present invention;

[0045] Figure 3 This is a sample diagram of splitting the tasks according to the layers of the model in a preferred embodiment of the large-scale distributed training method for a cloud environment of the present invention;

[0046] Figure 4 Schematic diagram of model deployment and training in a preferred embodiment of the large-scale distributed training method for cloud environments of the present invention;

[0047] Figure 5 Schematic diagram of step S400 in the large-scale distributed training method for cloud environments of the present invention;

[0048] Figure 6 Schematic diagram of step S500 in the large-scale distributed training method for cloud environments of the present invention;

[0049] Figure 7 This is a schematic diagram of a preferred embodiment of the large-scale distributed training system for cloud environments of the present invention;

[0050] Figure 8 FIG. 4 is a diagram showing the operating environment structure of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] The large-scale distributed training method for cloud environments described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the large-scale distributed training method for cloud environments includes the following steps:

[0053] Step S100: Upload the task, data and model to be trained to the task node, set the resources required for the task, and apply for several training nodes from the platform according to the resources required for the task.

[0054] Specifically, the user uploads the tasks, training data, and models to be trained to the task node, and sets the resources required to train the tasks; the resources include computing power, video memory, memory, network bandwidth, etc.

[0055] Computing power here refers to AI computing power, that is, floating-point computing power for matrix calculations.

[0056] Video memory, also known as frame buffer, is used to store rendering data that has been processed or is about to be retrieved by the graphics card chip. Like a computer's memory, video memory is a component used to store graphics information to be processed.

[0057] Memory is an important component of the computer, also known as internal memory and main memory. It is used to temporarily store the calculation data in the CPU and exchange data with external memory such as the hard disk.

[0058] Network bandwidth refers to the amount of data that can be transmitted per unit of time (usually one second). A network is similar to a highway: the greater the bandwidth, the more lanes a highway has, and the greater its capacity.

[0059] Furthermore, according to the computing power, video memory, internal memory and network bandwidth resources required for the task, several training nodes are applied to the platform according to actual needs. The training nodes contain resources such as computing power, video memory, internal memory and network bandwidth. Among them, the platform can be an intelligent computing network platform or a management platform of each computing power center.

[0060] The intelligent computing network platform or the management platform of each computing power center must also implement the management of security, privacy and resource scheduling, be responsible for ensuring the security of user data during cross-center transmission, be responsible for obtaining computing power resources from different computing power centers, and manage the obtained computing power resources.

[0061] It should be noted that the present invention can also upload a trained model and continue training the trained model.

[0062] Step S200: According to the resources required for the task and the resources in the several training nodes, the task is split into subtasks according to the hierarchy of the model, the subtasks are assigned to the several training nodes, and the training process of the model is set.

[0063] like Figure 3 As shown, the present invention splits the model parameters into three parts according to the model hierarchy of the model, breaks down the training task into three subtasks, and distributes them to three different training nodes for training.

[0064] Deep learning is a specialized machine learning method that utilizes deep neural networks to learn models. Models constructed using deep learning methods are called deep models. The learning process of a deep model can be viewed as learning a representation function Y = F(WX) for known data from a given dataset {X, Y}, where W is the model parameter to be learned. Deep models generally consist of three neural network layers: an input layer, a hidden layer, and an output layer. Each layer can contain one or more neural network layers, and each model layer can be viewed as a representation function. For example, the representation function Y = F(WX) of a deep model can be viewed as a cascade of representation functions from multiple model layers.

[0065] Specifically, the subtasks are divided according to the amount of computation required by the model hierarchy and the computing resources requested from the platform, so that the subtasks assigned to each training node take as much time as possible during the model training process, which is conducive to the efficient use of computing resources when executing parallel training tasks.

[0066] Furthermore, when decomposing the model hierarchically, it is also necessary to consider the scale of data transferred between model layers. The present invention selects the model splitting location where the scale of intermediate data transferred between layers is the smallest, thereby reducing the amount of data transmission between cross-domain computing centers.

[0067] It can be seen that the present invention decomposes the model into different subtasks according to the model hierarchy and assigns them to different training nodes. Each training node is only responsible for training part of the model parameters. There is no need to synchronize the model parameters during the training process. Each training node can come from different computing power centers in different regions, so that the user's model training task is no longer limited by the computing power of a single computing power center. The computing power resources distributed in computing power centers across the country can be used to complete the training tasks of larger-scale models, and support the use of computing power resources across multiple computing power centers to complete the training tasks of large-scale models.

[0068] After allocating the subtasks to the plurality of training nodes, the method further includes: setting a training process for the model.

[0069] Optimizing and setting the model training process based on the resource conditions of the several training nodes and the scale of the model parameters, wherein the optimization includes parallel optimization and communication optimization, and the setting includes setting the end condition and the number of iteration rounds;

[0070] The parallel optimization refers to selecting a parallel training strategy based on the computing power and network bandwidth of the task and the several training nodes.

[0071] Specifically, the current deep learning-based model training process includes forward computation and backpropagation. Calculations between different model layers are performed serially, requiring the previous layer to complete before the current layer can proceed. When subtasks at different layers of the model are assigned to different devices, only one device is actively performing the computation, while the other devices are in a waiting state, resulting in very low computing resource utilization. When using cross-domain computing power training, the transmission of intermediate variables between nodes requires a certain amount of time due to network transmission latency. While waiting for data transmission, the computing power of both the sender and receiver is idle, further reducing computing resource utilization.

[0072] To improve computing resource utilization and accelerate model training, the present invention employs a pipeline-like parallel approach between subtasks of different training nodes, based on the computing power and network bandwidth of the task and the training nodes. During execution, the original batch of data is split into smaller batches (m mini-batches), with each batch undergoing forward computation and backward propagation. After the previous mini-batch is calculated and while intermediate variables are being sent, the next mini-batch begins. After all mini-batches are calculated, the model parameters are updated uniformly.

[0073] Within the computing nodes, based on the network structure of the subtask model, you can choose to use parallel training methods such as data parallelism and model parallelism according to the task type. The entire model training process uses a hybrid parallel method including pipeline parallelism, model parallelism, and data parallelism to improve training speed.

[0074] The communication optimization refers to using a compression algorithm that meets the model accuracy loss requirements for the transmission data between the several training nodes.

[0075] Specifically, after the model parameters are decomposed according to the model hierarchy, the training tasks of the model parameters of different model layers are assigned to different training nodes across domains. The amount of data transmission between training nodes is related to the size of each batch of data (B), the sequence length (S), and the dimension of the hidden layer (H). The data scale Data_size that needs to be transferred is expressed as: Data size = B×S×H.

[0076] After the user task is set, the input sequence length and hidden layer dimension of the model are fixed. The data transmission scale can be controlled by adjusting the size of B, but the adjustment ability is limited, and the value of B is reduced to affect the iteration speed of model training on the entire data set. In order to further reduce the data transmission scale between nodes, the present invention uses the method of compressing the intermediate data transmitted between model layers to reduce the scale of cross-domain data transmission. The data compression methods that can be selected include quantization compression, low-rank decomposition, and sparsification. Among them, quantization compression is to use fewer bits to represent the target data, thereby reducing the total bit amount. For example, 8-bit quantization only uses 8 bits to represent the original 32-bit floating point number, and the transmission amount becomes 1 / 4 of the original; low-rank decomposition is to extract high-weight features through matrix decomposition to reduce the number of variables to be transmitted; sparsification is to reduce the number of variables to be transmitted by extracting a part of the variables according to a certain strategy. Different compression methods can be used in combination, such as first performing low-rank decomposition and then quantizing and compressing the results to further improve the compression rate and reduce the total bit amount that needs to be transmitted.

[0077] During the actual execution process, the appropriate compression rate can be selected based on factors such as the user task's requirements for accuracy loss, the network bandwidth of the computing environment, and other factors, so as to compress the size of the data to be transmitted as much as possible while maintaining the minimum loss of model accuracy.

[0078] Step S300: According to the training process, the data is split into several batches of data, and each batch of data is split into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node. Each mini-batch is sent to the first training node, and the data label corresponding to each mini-batch is sent to the last training node.

[0079] Specifically, in order to optimize the training process of the model in parallel, the present invention splits the original batch of data into smaller batches, splits the data into several batches of data, and splits each batch of data into m mini-batches. Each mini-batch of data is trained on a different training node, where the batch size can be determined by the actual hardware environment, such as video memory.

[0080] The present invention splits the training data into several batches of data, and further splits each batch of data into m mini-batches, that is, splitting a batch of data into m smaller data blocks for calculation. The purpose of this splitting is to improve the parallelism of the forward and backward calculation processes on each training node. By splitting each batch of data into m mini-batches, each training node can be better utilized, avoiding the problem of one training node performing a large amount of training while the other nodes are idle, and improving the efficiency of the entire training process. At the same time, splitting a batch of data into m steps for calculation reduces bandwidth consumption, providing a solution to the problem of lack of high-bandwidth network connection in cross-domain computing power centers.

[0081] Further, if Figure 4 As shown in the figure, after splitting the original batch of data into smaller batches, the training process is initialized, each mini-batch is sent to the first training node, and the data label corresponding to each mini-batch is sent to the last training node.

[0082] Step S400: Perform forward calculation on each mini-batch to obtain a calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain a reverse gradient, and fuse the reverse gradients corresponding to each mini-batch to obtain a target gradient. Update the model parameters on each training node according to the target gradient until the data training of each batch is completed.

[0083] like Figure 5 As shown, step S400 includes:

[0084] S410: Perform forward calculation on each mini-batch to obtain calculation results.

[0085] Specifically, each mini-batch is forward calculated on the first training node, and the first intermediate result output by the forward calculation is compressed using the compression algorithm and sent to the second training node. After receiving the first intermediate result, the second training node decompresses the first intermediate result; the decompressed first intermediate result is further forward calculated on the second training node, and the calculated second intermediate result is compressed using the compression algorithm and sent to the third training node, and calculations are performed in sequence according to the model hierarchy until the last training node obtains the calculation result.

[0086] S420: Perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain a reverse gradient.

[0087] Specifically, according to the calculation result at the last training node and the data label corresponding to the data, the loss is calculated by cross entropy or other methods, and the reverse gradient is calculated using the back propagation algorithm based on the calculated loss. The reverse gradient calculation result of the last training node is compressed using the compression algorithm and sent to the penultimate training node. The penultimate training node continues to perform reverse calculation and compresses and sends the calculation result, and the calculation is performed in sequence until the first training node obtains the reverse gradient of each mini-batch.

[0088] Cross entropy is an important concept in Shannon information theory, which is mainly used to measure the difference information between two probability distributions.

[0089] S430: Fuse the reverse gradients corresponding to each mini-batch to obtain the target gradient.

[0090] The reverse gradient corresponding to each mini-batch obtained by fusion calculation is adopted using methods such as gradient averaging to obtain the target gradient.

[0091] S440: Update the model parameters on each training node according to the target gradient until the data training of each batch is completed.

[0092] The process of fusing the reverse gradient and updating the model parameters on each training node are both performed on the training node corresponding to the subtask, without the need to synchronize model parameters across computing centers.

[0093] Step S500: Split the evaluation data into several batches of data, split each batch of data into m mini-batches, input each mini-batch into the training node after updating the model parameters for calculation, and obtain the final calculation result. Evaluate the final calculation result according to the data label corresponding to each mini-batch to obtain the evaluation result, until the evaluation of each batch of data is completed, and obtain the overall evaluation result based on all the evaluation results. If the overall evaluation result meets the end condition or number of iterations of model training, the training task is terminated.

[0094] like Figure 6 As shown, step S500 includes:

[0095] S510: Split the evaluation data into several batches of data, split each batch of data into m mini-batches, input each mini-batch into the training node after updating the model parameters for calculation, and obtain the final calculation result.

[0096] After the model parameters on each training node are updated, the training of the model is completed, and further, the quality of the model is evaluated.

[0097] Specifically, the prepared evaluation data is input into the training node after the model parameters are updated for calculation: the evaluation data is split into several batches of data, each batch of data is split into m mini-batches, and the forward calculation of each mini-batch is completed at the first training node after the model parameters are updated; the intermediate result output by the forward calculation is compressed using the compression algorithm and sent to the second training node, and the calculation is performed in sequence according to the model hierarchy until the last training node after the model parameters are updated to obtain the final calculation result.

[0098] S520 , evaluating the final calculation result according to the data label corresponding to each mini-batch to obtain an evaluation result, until the data evaluation of each batch is completed.

[0099] Specifically, at the last training node after the model parameters are updated, the final calculation result is evaluated according to the data label corresponding to each mini-batch, and the evaluation result of the model on each mini-batch is calculated until the data evaluation of each batch is completed.

[0100] S530. Obtain an overall evaluation result based on all evaluation results.

[0101] S540: If the overall evaluation result meets the end condition or number of iterations of model training, the training task is terminated.

[0102] Specifically, the overall evaluation result is compared with the set end conditions to determine whether the overall evaluation result meets the end conditions and whether the number of iterations has reached the preset number. If any of the conditions is met, the training task is terminated, where the number of iterations refers to the number of rounds required to complete model training using all training data.

[0103] If the overall evaluation result does not meet the end conditions or number of iterations of model training, the model will continue to be trained and evaluated.

[0104] Furthermore, if Figure 7 As shown, based on the above-mentioned large-model distributed training method for an inter-cloud environment, the present invention also provides a large-model distributed training system for an inter-cloud environment, wherein the large-model distributed training system for an inter-cloud environment includes:

[0105] Information preparation module 51: upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task;

[0106] Task splitting module 52: splits the task according to the resources required by the task and the resources in the training nodes according to the hierarchy of the model to obtain subtasks, assigns the subtasks to the training nodes, and sets the training process of the model;

[0107] Data splitting module 53: Splits the data into several batches according to the set training process, splits each batch into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, sends each mini-batch to the first training node, and sends the data label corresponding to each mini-batch to the last training node;

[0108] Model training module 54: performs forward calculation on each mini-batch to obtain a calculation result, performs reverse calculation based on the calculation result and the data label corresponding to the data to obtain a reverse gradient, fuses each of the reverse gradients to obtain a target gradient, and updates the model parameters on each training node based on the target gradient until the data training of each batch is completed;

[0109] Model evaluation module 55: Split the evaluation data into several batches of data, split each batch of data into m mini-batches, input each mini-batch into the training node after updating the model parameters for calculation, and obtain the final calculation result. Evaluate the final calculation result according to the data label corresponding to each mini-batch to obtain the evaluation result, until the evaluation of each batch of data is completed, and obtain the overall evaluation result based on all the evaluation results. If the overall evaluation result meets the end condition of model training or the number of iterations, the training task is terminated.

[0110] Furthermore, if Figure 8 As shown, based on the above-mentioned large-model distributed training method and system for cloud environments, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0111] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a large model distributed training program 40 for an inter-cloud environment is stored on the memory 20, and the large model distributed training program 40 for an inter-cloud environment can be executed by the processor 10, thereby realizing the large model distributed training method for an inter-cloud environment in the present application.

[0112] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes stored in the memory 20 or process data.

[0113] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0114] In one embodiment, when the processor 10 executes the large-model distributed training program 40 for the cloud environment in the memory 20, the following steps are implemented:

[0115] Upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task;

[0116] According to the resources required for the task and the resources in the several training nodes, the task is split into subtasks according to the hierarchy of the model, the subtasks are correspondingly assigned to the several training nodes, and a training process of the model is set;

[0117] According to the training process, the data is split into several batches of data, each batch of data is split into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, each mini-batch is sent to the first training node, and the data label corresponding to each mini-batch is sent to the last training node;

[0118] Perform forward calculation on each mini-batch to obtain the calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and fuse the reverse gradient corresponding to each mini-batch to obtain the target gradient. Update the model parameters on each training node according to the target gradient until the data training of each batch is completed;

[0119] The evaluation data is split into several batches of data, each batch of data is split into m mini-batches, each mini-batch is input into the training node after the model parameters are updated for calculation, and the final calculation result is obtained. The final calculation result is evaluated according to the data label corresponding to each mini-batch to obtain the evaluation result, until the evaluation of each batch of data is completed, and the overall evaluation result is obtained based on all the evaluation results. If the overall evaluation result meets the end condition of model training or the number of iterations, the training task is terminated.

[0120] Optionally, in the large-model distributed training method for cloud environments, the resources required for the task and the resources in the several training nodes include: computing power, video memory, internal memory and network bandwidth.

[0121] Optionally, the large-model distributed training method for a cloud environment, wherein the task is split according to the resources required for the task and the resources in the plurality of training nodes according to the hierarchy of the model to obtain subtasks, specifically includes:

[0122] Splitting the training task of the model parameters according to the model hierarchy of the model to obtain subtasks according to the resources required by the training task and the resources in the plurality of training nodes;

[0123] The model is split at the location where the scale of intermediate data transferred between layers is the smallest.

[0124] Optionally, in the large-scale distributed training method for a cloud environment, the process of setting up the training of the model specifically includes:

[0125] Optimizing and setting the model training process based on the resource conditions of the several training nodes and the scale of the model parameters, wherein the optimization includes parallel optimization and communication optimization, and the setting includes setting the end condition and the number of iteration rounds;

[0126] Among them, the parallel optimization refers to selecting a parallel training strategy based on the computing power and network bandwidth of the task and the several training nodes; the communication optimization refers to using a compression algorithm that meets the model accuracy loss requirements for the transmission data between the several training nodes.

[0127] Optionally, the large-model distributed training method for the cloud environment, wherein the forward calculation of each mini-batch to obtain the calculation result specifically includes:

[0128] Performing forward calculation on each mini-batch on the first training node, compressing a first intermediate result output by the forward calculation using the compression algorithm and sending the result to the second training node, and decompressing the first intermediate result after receiving the first intermediate result at the second training node;

[0129] The decompressed first intermediate result is forward-calculated on the second training node, and the calculated second intermediate result is compressed using the compression algorithm and sent to the third training node. The calculation is performed in sequence according to the model level until the calculation result is obtained at the last training node.

[0130] Optionally, the large-scale distributed training method for cloud environments, wherein the reverse calculation is performed based on the calculation results and the data labels corresponding to the data to obtain the reverse gradient, and each of the reverse gradients is fused to obtain the target gradient, specifically including:

[0131] Calculate the loss based on the calculation result at the last training node and the data label corresponding to the data, calculate the reverse gradient using the back propagation algorithm based on the calculated loss, compress the reverse gradient calculation result of the last training node using the compression algorithm and send it to the penultimate training node, the penultimate training node continues to perform reverse calculation and compresses and sends the calculation result, and calculates in sequence until the first training node obtains the reverse gradient of each mini-batch;

[0132] The reverse gradient of each mini-batch is fused to obtain the target gradient.

[0133] Optionally, the large-scale distributed training method for cloud environments further includes:

[0134] The process of fusing the reverse gradient and the process of updating the model parameters on each training node are both performed on the training node corresponding to the subtask.

[0135] Optionally, the large-model distributed training method for an inter-cloud environment, wherein the step of inputting each mini-batch into a training node after updating the model parameters for calculation to obtain a final calculation result, specifically includes:

[0136] Complete the forward calculation for the first training node of each mini-batch after updating the model parameters;

[0137] The intermediate result output by the forward calculation is compressed using the compression algorithm and sent to the second training node. The calculation is performed in sequence according to the model level until the last training node after the model parameters are updated to obtain the final calculation result.

[0138] Optionally, the large-scale distributed training method for cloud environments, wherein the step of evaluating the final calculation result according to the data labels corresponding to each mini-batch to obtain the evaluation result, specifically includes:

[0139] At the last training node after the model parameters are updated, the final calculation result is evaluated according to the data label corresponding to each mini-batch, and the evaluation result of the model on each mini-batch is calculated.

[0140] Optionally, the large-scale distributed training method for a cloud environment further includes:

[0141] If the overall evaluation result does not meet the end conditions or number of iterations of model training, the model will continue to be trained and evaluated.

[0142] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a large-model distributed training program for an inter-cloud environment, and when the large-model distributed training program for an inter-cloud environment is executed by a processor, the steps of the large-model distributed training method for an inter-cloud environment as described above are implemented.

[0143] In summary, the present invention aims to meet the scenario requirements of using cross-regional computing power centers for deep model training, and designs a deep model training method for cloud environments. It adopts strategies such as splitting model parameters according to network levels to different computing power centers, using compressed communication between cross-domain computing power nodes, and using hybrid parallelism within and between computing power centers to achieve the collaborative completion of deep model training using cross-domain computing power. The present invention can make large model training no longer restricted by the computing power resources of a single computing power center, and can efficiently utilize the computing power of cross-domain computing power centers, and use cross-domain computing power to collaboratively complete large-scale model training tasks in a low-bandwidth environment.

[0144] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0145] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0146] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A large-scale distributed training method for cloud environments, characterized by: The large-scale distributed training method for cloud environments includes: Upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task; According to the resources required for the task and the resources in the several training nodes, the task is split into subtasks according to the hierarchy of the model, the subtasks are correspondingly assigned to the several training nodes, and a training process of the model is set; According to the training process, the data is split into several batches of data, each batch of data is split into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, each mini-batch is sent to the first training node, and the data label corresponding to each mini-batch is sent to the last training node; Perform forward calculation on each mini-batch to obtain the calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and fuse the reverse gradient corresponding to each mini-batch to obtain the target gradient. Update the model parameters on each training node according to the target gradient until the data training of each batch is completed; The evaluation data is split into several batches of data, each batch of data is split into m mini-batches, each mini-batch is input into the training node after the model parameters are updated for calculation, and the final calculation result is obtained. The final calculation result is evaluated according to the data label corresponding to each mini-batch to obtain the evaluation result, until the evaluation of each batch of data is completed, and the overall evaluation result is obtained based on all the evaluation results. If the overall evaluation result meets the end condition of model training or the number of iterations, the training task is terminated.

2. The large-scale distributed training method for cloud environments according to claim 1 is characterized in that: The resources required for the task and the resources in the several training nodes include: computing power, video memory, internal memory and network bandwidth.

3. The large-scale distributed training method for cloud environments according to claim 1 is characterized in that: The task is split according to the resources required by the task and the resources in the training nodes according to the hierarchy of the model to obtain subtasks, specifically including: Splitting the training task of the model parameters according to the model hierarchy of the model to obtain subtasks according to the resources required by the training task and the resources in the plurality of training nodes; The model is split at the location where the scale of intermediate data transferred between layers is the smallest.

4. The large-scale distributed training method for cloud environments according to claim 2 is characterized in that: The training process of setting the model specifically includes: Optimizing and setting the model training process based on the resource conditions of the several training nodes and the scale of the model parameters, wherein the optimization includes parallel optimization and communication optimization, and the setting includes setting the end condition and the number of iteration rounds; Among them, the parallel optimization refers to selecting a parallel training strategy based on the computing power and network bandwidth of the task and the several training nodes; the communication optimization refers to using a compression algorithm that meets the model accuracy loss requirements for the transmission data between the several training nodes.

5. The large-scale distributed training method for cloud environments according to claim 4 is characterized in that: The forward calculation is performed on each mini-batch to obtain the calculation results, which specifically include: Performing forward calculation on each mini-batch on the first training node, compressing a first intermediate result output by the forward calculation using the compression algorithm and sending the result to the second training node, and decompressing the first intermediate result after receiving the first intermediate result at the second training node; The decompressed first intermediate result is forward-calculated on the second training node, and the calculated second intermediate result is compressed using the compression algorithm and sent to the third training node. The calculation is performed in sequence according to the model level until the calculation result is obtained at the last training node.

6. The large-scale distributed training method for cloud environments according to claim 5 is characterized in that: The reverse calculation is performed based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and each of the reverse gradients is fused to obtain the target gradient, specifically including: Calculate the loss based on the calculation result at the last training node and the data label corresponding to the data, calculate the reverse gradient using the back propagation algorithm based on the calculated loss, compress the reverse gradient calculation result of the last training node using the compression algorithm and send it to the penultimate training node, the penultimate training node continues to perform reverse calculation and compresses and sends the calculation result, and calculates in sequence until the first training node obtains the reverse gradient of each mini-batch; The reverse gradient of each mini-batch is fused to obtain the target gradient.

7. The large-scale distributed training method for cloud environments according to claim 1 is characterized in that: The large-scale distributed training method for cloud environments also includes: The process of fusing the reverse gradient and the process of updating the model parameters on each training node are both performed on the training node corresponding to the subtask.

8. The large-scale distributed training method for cloud environments according to claim 1 is characterized in that: The aforementioned step of inputting each mini-batch into the training node after updating the model parameters to perform calculations and obtain the final calculation results specifically includes: Complete the forward calculation for the first training node of each mini-batch after updating the model parameters; The intermediate results of the forward calculation output are compressed using a compression algorithm and sent to the second training node. The calculations are performed sequentially according to the model level until the last training node after the model parameters are updated to obtain the final calculation results.

9. The large-scale distributed training method for cloud environments according to claim 8 is characterized in that: The final calculation result is evaluated according to the data label corresponding to each mini-batch to obtain an evaluation result, specifically including: At the last training node after the model parameters are updated, the final calculation result is evaluated according to the data labels corresponding to each mini-batch, and the evaluation result of the model on each mini-batch is calculated.

10. The large-scale distributed training method for cloud environments according to claim 1 is characterized in that: The large-scale distributed training method for cloud environments further includes: If the overall evaluation result does not meet the end conditions or number of iterations of model training, the model will continue to be trained and evaluated.

11. A large-scale distributed training system for cloud environments, characterized by: The large-scale distributed training system for cloud environments includes: Information preparation module: upload the task, data, and model to be trained to the task node, set the resources required for the task, and apply for a number of training nodes from the platform based on the resources required for the task; Task splitting module: splits the task according to the resources required by the task and the resources in the training nodes according to the hierarchy of the model to obtain subtasks, assigns the subtasks to the training nodes, and sets the training process of the model; Data splitting module: Split the data into several batches according to the set training process, split each batch of data into m mini-batches, where m is the number of segments of the batch data for pipeline parallel computing at each training node, send each mini-batch to the first training node, and send the data label corresponding to each mini-batch to the last training node; Model training module: Perform forward calculation on each mini-batch to obtain the calculation result, perform reverse calculation based on the calculation result and the data label corresponding to the data to obtain the reverse gradient, and fuse each of the reverse gradients to obtain the target gradient. Based on the target gradient, the model parameters on each training node are updated until the data training of each batch is completed; Model evaluation module: Split the evaluation data into several batches of data, split each batch of data into m mini-batches, input each mini-batch into the training node after updating the model parameters for calculation, and obtain the final calculation result. Evaluate the final calculation result according to the data label corresponding to each mini-batch to obtain the evaluation result until the evaluation of each batch of data is completed. Get the overall evaluation result based on all the evaluation results. If the overall evaluation result meets the end condition or number of iterations of model training, the training task is terminated.

12. A terminal, characterized in that: The terminal includes: a memory, a processor, and a large-model distributed training program for an inter-cloud environment stored in the memory and runnable on the processor. When the large-model distributed training program for an inter-cloud environment is executed by the processor, the steps of the large-model distributed training method for an inter-cloud environment as described in any one of claims 1-10 are implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a large-model distributed training program for an inter-cloud environment. When the large-model distributed training program for an inter-cloud environment is executed by a processor, the steps of the large-model distributed training method for an inter-cloud environment as described in any one of claims 1-10 are implemented.

Images