Model training method and device, electronic device, storage medium and program product

Abstract

The present disclosure relates to a model training method and device, electronic equipment, storage medium and program product, and relates to the technical field of computers. The method comprises: cutting a to-be-trained model into N parts and distributing the N parts to N second processors respectively, each second processor corresponding to a pipeline for training a different part of the to-be-trained model; and cooperatively scheduling a first processor and the N second processors to perform pipeline parallel training on the to-be-trained model to obtain a trained target model. The first processor is configured to jointly train a first part of the to-be-trained model with the first second processor and to jointly train an Nth part of the to-be-trained model with the Nth second processor, so that the training time of each second processor for different parts of the to-be-trained model is the same. The embodiments of the present disclosure can eliminate or reduce the idle waiting time of each second processor while maintaining the integrity and compatibility of the to-be-trained model, thereby improving the efficiency of model training.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a model training method and device, electronic device, storage medium and program product. BACKGROUND

[0002] With the rapid development of deep learning technology, especially in the fields of computer vision, natural language processing and speech recognition, the scale and complexity of neural network models grow exponentially. In the training process of large-scale neural network models, pipeline parallel technology is widely used to improve the efficiency of model training. However, in pipeline parallel training, in order to wait for the completion of upstream calculation, some computing devices (such as GPU) will have idle time, thereby reducing the overall training efficiency. SUMMARY

[0003] The present disclosure provides a model training method and device, electronic device, storage medium and program product.

[0004] According to an aspect of the present disclosure, a model training method is provided, which is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors which are different from the first processor in type, the method comprising: dividing a to-be-trained model into N parts and assigning the N parts to the N second processors respectively, wherein a pipeline corresponding to an i-th second processor is used to train an i-th part of the to-be-trained model, each part of the to-be-trained model has the same number of network layers, i is an integer within 1 to N; and the first processor and the N second processors are cooperatively scheduled to perform pipeline parallel training on the to-be-trained model to obtain a trained target model; wherein the first processor is used to train a first part of the to-be-trained model together with a first second processor, and train an N-th part of the to-be-trained model together with an N-th second processor, and the time for the first second processor to train the first part of the to-be-trained model and the time for the N-th second processor to train the N-th part of the to-be-trained model are the same as the time for the remaining second processors to train the respective corresponding parts of the to-be-trained model.

[0005] In a possible implementation, the first processor and the N second processors are cooperatively scheduled to perform pipeline parallel training on the to-be-trained model to obtain a trained target model, including: for the first second processor, the first second processor and the first processor are scheduled to jointly perform forward calculation and reverse calculation on the first part of interlaces allocated to the first second processor to obtain forward calculation results and reverse calculation results of the first part; for any processor in the second second processor to the N-1th second processor, each ith processor is scheduled to perform forward calculation and reverse calculation on the ith part of interlaces allocated to the ith processor to obtain forward calculation results and reverse calculation results of the ith part; for the Nth second processor, the Nth second processor and the first processor are scheduled to jointly perform forward calculation and reverse calculation on the Nth part of interlaces allocated to the Nth second processor to obtain forward calculation results and reverse calculation results of the Nth part; and the trained target model is obtained according to N reverse calculation results corresponding to the N parts.

[0006] In a possible implementation, the first second processor and the first processor are scheduled to jointly perform forward calculation on the first part allocated to the first second processor to obtain forward calculation results of the first part, including: before the end of each round of distributed training of the to-be-trained model, the first processor is scheduled to pre-read training data of the next round according to the first part, and pre-process the training data of the next round to obtain pre-fetch data of the next round; and in response to starting distributed training of the to-be-trained model in the next round, the first second processor is scheduled to determine forward calculation results of the first part according to the pre-fetch data from the first processor.

[0007] In a possible implementation, the first second processor and the first processor are scheduled to jointly perform reverse calculation on the first part allocated to the first second processor to obtain reverse calculation results of the first part, including: the first second processor is scheduled to perform reverse calculation on the first M-K network layers in the first part to obtain first intermediate reverse calculation results; and the first processor is scheduled to perform reverse calculation on the last K network layers in the first part according to the first intermediate reverse calculation results to obtain reverse calculation results of the first part; wherein M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M.

[0008] In a possible implementation, the method further includes: obtaining a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of the first second processor for training the first part of the to-be-trained model; and adjusting a network layer number K of the first processor for performing back propagation in the first part of the to-be-trained model according to the first operation time and the second operation time.

[0009] In a possible implementation, the method further includes: obtaining a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of the first second processor for training the first part of the to-be-trained model; and adjusting a network layer number K of the first processor for performing back propagation in the first part of the to-be-trained model according to the first operation time and the second operation time.

[0010] In a possible implementation, the method further includes: obtaining a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of the first second processor for training the first part of the to-be-trained model; and adjusting a network layer number K of the first processor for performing back propagation in the first part of the to-be-trained model according to the first operation time and the second operation time.

[0011] In a possible implementation, the method further includes: obtaining a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of the first second processor for training the first part of the to-be-trained model; and adjusting a network layer number K of the first processor for performing back propagation in the first part of the to-be-trained model according to the first operation time and the second operation time.

[0012] According to an aspect of the present disclosure, a model training apparatus is provided, which comprises: the apparatus is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprises a first processor and N second processors which are different from the first processor, the apparatus comprises: a splitting module configured to split a to-be-trained model into N parts which are respectively allocated to the N second processors, wherein a pipeline of an i-th second processor is configured to train an i-th part of the to-be-trained model, each part of the to-be-trained model has a same number of network layers, i is an integer within 1 to N; a training module configured to cooperatively schedule the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model, and obtain a trained target model; wherein the first processor is configured to train a first part of the to-be-trained model together with a first second processor, and train an N-th part of the to-be-trained model together with an N-th second processor, and a time for the first second processor to train the first part of the to-be-trained model and a time for the N-th second processor to train the N-th part of the to-be-trained model are same as a time for the remaining second processors to train respective corresponding parts of the to-be-trained model.

[0013] In a possible implementation, the training module is configured to: for the first second processor, schedule the first second processor and the first processor to perform forward calculation and backward calculation on the first part allocated to the first second processor in an interleaving manner, to obtain a forward calculation result and a backward calculation result of the first part; for any processor in the second second processor to the N-1-th second processor, schedule each i-th processor to perform forward calculation and backward calculation on the i-th part allocated to the i-th processor in an interleaving manner, to obtain a forward calculation result and a backward calculation result of the i-th part; for the N-th second processor, schedule the N-th second processor and the first processor to perform forward calculation and backward calculation on the N-th part allocated to the N-th second processor in an interleaving manner, to obtain a forward calculation result and a backward calculation result of the N-th part; and obtain the trained target model according to N backward calculation results corresponding to the N parts.

[0014] In a possible implementation, the scheduling the first second processor and the first processor to jointly perform forward calculation on the first part to which the first second processor is allocated, to obtain a forward calculation result of the first part, comprises: before the end of each round of distributed training of the to-be-trained model, scheduling the first processor to pre-read training data of a next round according to the first part, and to pre-process the training data of the next round to obtain pre-fetch data of the next round; in response to starting the distributed training of the to-be-trained model of the next round, scheduling the first second processor to determine the forward calculation result of the first part according to the pre-fetch data from the first processor.

[0015] In a possible implementation, the scheduling the first second processor and the first processor to jointly perform backward calculation on the first part to which the first second processor is allocated, to obtain a backward calculation result of the first part, comprises: scheduling the first second processor to perform backward calculation on the first M-K network layers in the first part, to obtain a first intermediate backward calculation result; and scheduling the first processor to perform backward calculation on the last K network layers in the first part according to the first intermediate backward calculation result, to obtain the backward calculation result of the first part; wherein M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M.

[0016] In a possible implementation, the apparatus further comprises an adjusting module configured to: obtain a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of the first second processor for training the first part of the to-be-trained model; and adjust the number K of network layers in which the first processor performs backward calculation in the first part of the to-be-trained model, according to the first operation time and the second operation time.

[0017] In a possible implementation, the scheduling the Nth second processor and the first processor to jointly perform forward calculation on the Nth part to which the Nth second processor is allocated, to obtain a forward calculation result of the Nth part, comprises: scheduling the Nth second processor to perform forward calculation on the Nth part, to obtain a prediction result output by the Nth part; and scheduling the first processor to determine a loss value according to the prediction result and a preset loss function, and to take the loss value as the forward calculation result of the Nth part, wherein the loss value is used to guide the first processor and the N second processors to perform backward calculation.

[0018] In a possible implementation, the scheduling the Nth second processor and the first processor to jointly perform back propagation calculation on the Nth part to which the Nth second processor is assigned to obtain a back propagation calculation result of the Nth part comprises: scheduling the Nth second processor to perform back propagation calculation on the first M-K network layers in the Nth part to obtain a second intermediate back propagation calculation result; and scheduling the first processor to perform back propagation calculation on the last K network layers in the Nth part according to the second intermediate back propagation calculation result to obtain the back propagation calculation result of the Nth part; wherein M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M.

[0019] In a possible implementation, the adjusting module is further configured to: obtain a third operation time of the first processor in training the Nth part of the to-be-trained model and a fourth operation time of the Nth second processor in training the Nth part of the to-be-trained model; and adjust the number K of network layers in which the first processor performs back propagation calculation on the Nth part of the to-be-trained model according to the third operation time and the fourth operation time.

[0020] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

[0021] According to an aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0022] According to an aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0023] The model training method of the embodiments of the present disclosure can be applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors different from the first processor, the method comprising: dividing a to-be-trained model into N parts and distributing the N parts to the N second processors respectively, wherein a pipeline of an i-th second processor is used to train an i-th part of the to-be-trained model, each part of the to-be-trained model has the same number of network layers, and i is an integer within 1 to N; and cooperatively scheduling the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model to obtain a trained target model; wherein the first processor is used to train a first part of the to-be-trained model together with a first second processor and train an Nth part of the to-be-trained model together with an Nth second processor, and a time for the first second processor to train the first part of the to-be-trained model and a time for the Nth second processor to train the Nth part of the to-be-trained model are the same as a time for the remaining second processors to train respective corresponding parts of the to-be-trained model.

[0024] Thus, this mechanism of cooperative work of heterogeneous computing resources (such as the first processor and the second processor) optimizes the utilization rate of computing resources in the pipeline parallel training process, and by executing part of the computing task on the first processor, the training time of each second processor for each part of the to-be-trained model allocated thereto is the same, thereby eliminating or reducing the idle waiting time caused by computing dependency in the model training process, and improving the efficiency of model training.

[0025] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] Figure 1 A flowchart of a model training method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram of a distributed training method according to an embodiment of the present disclosure is shown.

[0029] Figure 3 An effect comparison schematic diagram of a model training method according to an embodiment of the present disclosure is shown.

[0030] Figure 4A block diagram of a model training apparatus according to an embodiment of the disclosure is shown.

[0031] Figure 5 A block diagram of an electronic device according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0032] Various exemplary embodiments, features, and aspects of the disclosure will be explained in greater detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote the same elements or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0033] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0034] The term "and / or" used herein is merely used to describe associated objects, and can represent three meanings, for example, A and / or B can mean that there are A alone, A and B together, and B alone. In addition, the term "at least one of" used herein means any one of the plurality of or any combination of at least two of the plurality of, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0035] In addition, in order to better illustrate the disclosure, numerous specific details are given in the specific embodiments below. Those skilled in the art will understand that the disclosure can also be implemented without certain specific details. In some examples, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail in order to highlight the main idea of the disclosure.

[0036] In related technologies, in a pipeline parallel design, different computing devices (such as GPUs) will be in an idle state when waiting for dependent computations to complete, resulting in waste of computing resources and reduction of training efficiency, especially in large-scale neural network model training. In order to optimize the computing efficiency in the pipeline parallel processing process, one network layer (such as a Transformer layer) can be reduced at both ends of the pipeline to reduce the idle waiting time caused by computational dependencies, and a more compact pipeline parallel structure is attempted to be implemented to reduce or eliminate the idle waiting time.

[0037] This method in related technologies is to directly simplify the computing task by reducing the number of network layers (such as Transformer layers) at both ends of the pipeline parallel training, thereby reducing the waiting time and improving the training efficiency. This method, although simple to operate, can directly reduce the computing load, thereby reducing or eliminating the idle waiting phenomenon to a certain extent.

[0038] However, by reducing the number of network layers (e.g., Transformer layers), although the calculation can be simplified and the idle waiting time can be reduced, this method sacrifices the integrity of the neural network model, which may lead to performance degradation, especially when integrated with mainstream open source model structures, compatibility problems may occur. This is because most neural network models and frameworks are designed based on complete network layers (e.g., Transformer layers), and any reduction in the number of layers may result in suboptimal performance of the neural network model, or the need for significant adjustments to the neural network model architecture.

[0039] Therefore, although simplifying the number of layers of the neural network model is a viable alternative, its limitations in practical applications are apparent, especially in scenarios where optimal model performance and compatibility with deep learning frameworks are pursued. Reducing the number of network layers (e.g., Transformer layers) affects the performance of the neural network model and makes it difficult to be compatible with mainstream open source neural network model structures.

[0040] Moreover, the utilization of computing resources in the pipeline parallel process in the related art is not balanced, mainly relying on high-performance computing devices such as GPUs, and less relying on other computing resources such as CPUs, the participation of CPU resources in this process is extremely low, leading to insufficient utilization of computing resources and waste of computing resources. In view of this, the embodiments of the present disclosure provide a model training method that can eliminate or reduce idle waiting time, fully utilize computing resources, and improve the efficiency of model training while maintaining the integrity and compatibility of the model to be trained.

[0041] Figure 1 A flowchart of a model training method according to an embodiment of the present disclosure is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors different in type from the first processor, the method comprising:

[0042] In step S11, the model to be trained is divided into N parts and allocated to the N second processors respectively, wherein the pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained, each part of the model to be trained has the same number of network layers, and i is an integer within 1 to N;

[0043] In step S12, the first processor and the N second processors are cooperatively scheduled to train the model to be trained in pipeline parallel, and a trained target model is obtained;

[0044] The first processor is configured to jointly train a first part of the to-be-trained model with a first second processor and jointly train an Nth part of the to-be-trained model with an Nth second processor, and the first second processor trains the first part of the to-be-trained model for the same time as the Nth second processor trains the Nth part of the to-be-trained model, and the time for the remaining second processors to train respective corresponding parts of the to-be-trained model is the same.

[0045] In a possible implementation, the heterogeneous system is composed of processors of different architectures, and the heterogeneous system is an electronic device, for example, a server, a user equipment (UE), a mobile device, a user terminal, a terminal device, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like. Embodiments of the present disclosure do not limit the specific form of the heterogeneous system.

[0046] In a possible implementation, the model training method can be executed by a dynamic scheduler. The dynamic scheduler can be a processor running preset software or preset program code in the heterogeneous system. The preset software or preset program code is used to dynamically schedule other processors, and can be implemented by a hardware co-description language, an assembly language, a high-level language (for example, C, C++, and the like), a script language, or the like. Alternatively, the dynamic scheduler can also be a logic circuit embedded in the heterogeneous system. Embodiments of the present disclosure do not limit the form of the dynamic scheduler.

[0047] In a possible implementation, the first processor in the heterogeneous system can include, but is not limited to, a central processing unit (CPU), a digital signal processing unit (DSP), an application specific integrated circuit (ASIC), a tensor processing unit (TPU), a field programmable gate array (FPGA), and the like.

[0048] In a possible implementation, the second processor in the heterogeneous system can include, but is not limited to, a Graphic Processing Unit (GPU), a General-Purpose Computing on Graphics Processing Units (GPGPU), a Neural Network Processing Unit (NPU), and the like.

[0049] It should be understood that the first processor and the second processor are of different types, and embodiments of the present disclosure do not limit the specific types of the first processor and the second processor.

[0050] In a possible implementation, the model training method can be for a distributed training scenario, and the N second processors (for example, multi-core GPUs, GPUs with multiple stream processor units), and the second processor (for example, a CPU) can be interconnected and communicate. The model training method of the embodiments of the present disclosure can effectively allocate training tasks to the first processor (for example, a CPU) and the N second processors (for example, GPUs), thereby improving the speed and efficiency of distributed training.

[0051] In a possible implementation, the to-be-trained model is an artificial intelligence large model, for example, including a Transformer model, which is an attention mechanism-based deep learning architecture, and can include multiple network layers (for example, Transformer layers), each Transformer layer can be composed of an encoder (Encoder) and a decoder (Decoder), the encoder is responsible for extracting input sequence features, and the decoder is responsible for generating target sequences.

[0052] The to-be-trained model can include a forward phase and a backward phase in the training process. In the forward phase of the to-be-trained model, the training data can be sent into the to-be-trained model in batches, and the input training data can be converted into an output result by the to-be-trained model, which is a prediction of the to-be-trained model on the training data. Then, the to-be-trained model can call a loss function to calculate the loss value (for example, error) between the output result and the true label. In the backward phase of the to-be-trained model, the loss value obtained by the forward node can be used to update the weight parameters of the to-be-trained model, so that the output result of the to-be-trained model is closer to the true label. The training process of the to-be-trained model can include multiple rounds of training, each round of training can be composed of a forward phase and a backward phase, and the to-be-trained model can gradually learn how to extract useful features from the input training data and produce more accurate prediction results through multiple rounds of iterative training, until a trained target model is obtained.

[0053] In a possible implementation, for a to-be-trained model with a super-large scale parameter (e.g., tens of trillions), a single-machine single-card mode processor cannot meet the requirement of training the super-large to-be-trained model, because the to-be-trained model is too large to be placed in a single processor. Therefore, in step S11, the dynamic scheduler can perform a splitting operation on the to-be-trained model, and distribute different network layers of the to-be-trained model to N second processors, to reduce the memory consumption of a single computing device, so as to train the super-large to-be-trained model.

[0054] Figure 2 A schematic diagram of a distributed training method according to an embodiment of the present disclosure is shown as follows. Figure 2 As shown in the figure, the first pipeline corresponding to the first second processor is used to train the first part of the to-be-trained model, the second pipeline corresponding to the second second processor is used to train the second part of the to-be-trained model, and so on, and the Nth pipeline corresponding to the Nth second processor is used to train the Nth part of the to-be-trained model.

[0055] In each round of training, the first part of the to-be-trained model includes additional data acquisition and mapping operations of an embedding layer, and the Nth part of the to-be-trained model includes additional language model head (lm_head) calculation and loss calculation. If the first processor does not intervene in the training, the execution time of the first second processor and the Nth second processor is greater than that of the other second processors, and considering the calculation dependency relationship between the second processors, there is an idle waiting time.

[0056] Therefore, the calculation tasks of the first and last pipeline stages need to be redesigned. In step S12, the dynamic scheduler can schedule the first processor to assist the first second processor in training the first part of the to-be-trained model, and assist the Nth second processor in training the Nth part of the to-be-trained model. In this way, the dynamic scheduler can cooperatively schedule the first processor and the N second processors to jointly perform pipeline parallel training on the to-be-trained model, to obtain a trained target model. This mechanism of cooperative work of heterogeneous computing resources (such as the first processor and the second processor) optimizes the utilization rate of the computing resources in the pipeline parallel training process, and realizes an efficient training process. The mechanism can include but is not limited to data prefetching, performing part of the calculation tasks on the first processor to reduce the idle time of the second processor, and the like, so that the training time of each second processor for each part of the to-be-trained model allocated to the second processor is the same.

[0057] It can be known that the dynamic scheduler can reduce resource waste and improve the computing efficiency of the overall system by coordinating the scheduling of different types of computing resources (e.g., the first processor and the second processor). Moreover, without changing the structure of the to-be-trained model (e.g., keeping the original Transformer layer of the to-be-trained model), the first and last pipeline parallel stages can be redesigned to eliminate or reduce idle waiting time, thereby achieving structural compatibility for various to-be-trained models and reducing the impact on the performance of the to-be-trained model.

[0058] In this way, by introducing the first processor into the computing process at the beginning and end of the pipeline parallel stage, the idle waiting time caused by waiting for data or computing dependencies can be effectively eliminated, thereby significantly improving the continuity and efficiency of the pipeline parallel training process.

[0059] In a possible implementation, step S12 can include: for the first second processor, scheduling the first second processor and the first processor to jointly perform forward calculation and reverse calculation on the first part of the interlace allocated to the first second processor to obtain the forward calculation result and the reverse calculation result of the first part; for any one of the second second processor to the N-1th second processor, scheduling each ith processor to perform forward calculation and reverse calculation on the ith part of the interlace allocated to the ith processor to obtain the forward calculation result and the reverse calculation result of the ith part; for the Nth second processor, scheduling the Nth second processor and the first processor to jointly perform forward calculation and reverse calculation on the Nth part of the interlace allocated to the Nth second processor to obtain the forward calculation result and the reverse calculation result of the Nth part; and obtaining the trained target model according to the N reverse calculation results corresponding to the N parts.

[0060] For example, it is assumed that there are four (N = 4) second processors, i.e., the second processor 1 to the second processor 4, where the second processor 1 represents the first second processor, the second processor 2 represents the second second processor, the second processor 3 represents the third second processor, and the second processor 4 represents the fourth second processor.

[0061] The dynamic scheduler can schedule the first processor and the second processor 1 to jointly train the first part of the to-be-trained model, schedule the second processor 2 to train the second part of the to-be-trained model, schedule the second processor 3 to train the third part of the to-be-trained model, and schedule the first processor and the second processor 4 to jointly train the fourth part of the to-be-trained model.

[0062] For each round of training, the dynamic scheduler can schedule the first processor to assist the second processor 1 to perform forward calculation, calculate the output of the 1st part of the model to be trained, and pass the output as forward calculation result 1 to the second processor 2.

[0063] The dynamic scheduler can schedule the second processor 2 to receive the forward calculation result 1 output by the second processor 1, and perform forward calculation according to the forward calculation result 1, calculate the output of the 2nd part of the model to be trained, and pass the output as forward calculation result 2 to the second processor 3.

[0064] The dynamic scheduler can schedule the second processor 3 to receive the forward calculation result 2 output by the second processor 2, and perform forward calculation according to the forward calculation result 2, calculate the output of the 3rd part of the model to be trained, and pass the output as forward calculation result 3 to the second processor 4.

[0065] The dynamic scheduler can schedule the second processor 4 to receive the forward calculation result 3 output by the second processor 3, and schedule the first processor to assist the second processor 4 to jointly perform forward calculation according to the forward calculation result 3, calculate the output of the 4th part of the model to be trained, obtain the final model output of this round, and calculate the loss value according to the loss function, and pass the loss value as forward calculation result 4.

[0066] The dynamic scheduler can schedule the first processor to assist the second processor 4 to perform backward calculation according to the loss value, calculate the gradient 1 of the 4th part of the model to be trained, and update the parameters of the 4th part of the model to be trained using the gradient 1 as the backward calculation result 1, and then pass the gradient 1 and the loss value back to the second processor 3.

[0067] The dynamic scheduler can schedule the second processor 3 to receive the gradient 1 and the loss value output by the second processor 4, perform backward calculation according to the gradient 1 and the loss value, calculate the gradient 2 of the 3rd part of the model to be trained, and update the parameters of the 3rd part of the model to be trained using the gradient 2 as the backward calculation result 2, and then pass the gradient 2 and the loss value to the second processor 2.

[0068] The dynamic scheduler can schedule the second processor 2 to receive the gradient 2 and the loss value output by the second processor 3, perform backward calculation according to the gradient 2 and the loss value, calculate the gradient 3 of the 2nd part of the model to be trained, and update the parameters of the 2nd part of the model to be trained using the gradient 3 as the backward calculation result 3, and then pass the gradient 3 to the second processor 1.

[0069] The dynamic scheduler can schedule the second processor 1 to receive the gradient 3 and the loss value output by the second processor 2, and schedule the first processor to assist the second processor 1 to jointly perform the back calculation according to the gradient 3 and the loss value, calculate the gradient 4 of the first part of the to-be-trained model, and update the parameters of the first part of the to-be-trained model as the back calculation result 4 using the gradient 4.

[0070] Since the back calculation result of each part of the to-be-trained model is the parameter update result of each part, the corresponding back calculation result of each part is obtained, and the model training of this round is completed.

[0071] In order to improve the parallelism, each second processor can perform the forward calculation and the back calculation on the allocated part in an interleaved manner, wherein the forward calculation and the back calculation can be different training tasks of different rounds.

[0072] If different second processors take different time to perform the forward calculation (or the back calculation) on the allocated part, the second processors will be in an idle state when waiting for the completion of the dependent calculation, which reduces the model training efficiency.

[0073] Therefore, considering that the first part and the last Nth part of the to-be-trained model take more time to train, for the first second processor, the first second processor and the first processor can be scheduled to jointly perform the forward calculation and the back calculation on the first part allocated to the first second processor in an interleaved manner; for the Nth second processor, the Nth second processor and the first processor can be scheduled to jointly perform the forward calculation and the back calculation on the Nth part allocated to the Nth second processor in an interleaved manner, so that each second processor takes the same time to train each part of the to-be-trained model.

[0074] Figure 3 An effect comparison diagram of a model training method according to an embodiment of the present disclosure is shown.

[0075] As shown in Figure 3 The second processor 1 is responsible for training the first part of the to-be-trained model, the second processor 2 is responsible for training the second part of the to-be-trained model, the second processor 3 is responsible for training the third part of the to-be-trained model, and the second processor 4 is responsible for training the fourth part of the to-be-trained model. The numbers "1" to "8" represent the training rounds, for example, "1" represents the first round of training, "2" represents the second round of training, and so on, and "8" represents the eighth round of training. Each round of training includes forward calculation and back calculation. In the forward calculation of each round of training, the second processor 2 depends on the second processor 1, the second processor 3 depends on the second processor 2, and the second processor 4 depends on the second processor 3; in the back calculation of each round of training, the second processor 3 depends on the second processor 4, the second processor 2 depends on the second processor 3, and the second processor 1 depends on the second processor 2.

[0076] In the stable stage, each processor can perform forward calculation and backward calculation on the assigned part of the to-be-trained model in an interleaved manner, where the forward calculation and the backward calculation can be different rounds of training tasks. Figure 3 As shown in the figure, in the stable stage, the second processor 1 can participate in the backward calculation of the first round of training, the forward calculation of the fifth round of training, the backward calculation of the second round of training, the forward calculation of the sixth round of training, the backward calculation of the third round of training, the forward calculation of the seventh round of training, the backward calculation of the fourth round of training, and the forward calculation of the eighth round of training in sequence.

[0077] Figure 3 The upper half of the middle part is an effect diagram when the first processor does not participate in training. Because the second processor 1 takes more time to train the first part of the to-be-trained model and the second processor 4 takes more time to train the last fourth part of the to-be-trained model, there will be a lot of idle waiting time caused by calculation dependency even in the stable stage.

[0078] Figure 3 The lower half of the middle part is an effect diagram when the first processor participates in training. Because the first processor trains the first part of the to-be-trained model together with the second processor 1 and trains the fourth part of the to-be-trained model together with the second processor 4, so that the time for the second processor 1 to train the first part of the to-be-trained model and the time for the second processor 4 to train the fourth part of the to-be-trained model are the same as the time for the remaining two second processors 2 and 3 to train their respective parts, in the stable stage, the idle waiting time caused by waiting for data or calculation dependency can be effectively eliminated, thereby significantly improving the continuity and efficiency of the pipeline parallel training process.

[0079] It should be understood that, Figure 3 The specific interleaving order can be set according to the actual application scenario, and the embodiments of the present disclosure do not limit the interleaving order.

[0080] In a possible implementation, after the distributed training of the to-be-trained model is completed, a trained target model is obtained, the to-be-processed data is input into the target model for processing, and a processing result is obtained. The target model is a model used to perform at least one task of image classification, target detection, image segmentation, and speech recognition. The to-be-processed data includes at least one of image data, speech data, and text data. The target model is trained according to the model training method described in steps S11-S12. The trained target model can be applied to various inference tasks such as image classification, target detection, image segmentation, and speech recognition, and can efficiently and accurately determine the inference result of the inference task.

[0081] The model training method of the embodiments of the present disclosure is described below.

[0082] Before the distributed training starts, the dynamic scheduler can be initialized, for example, including configuring computing resources, initializing network connections, and loading training algorithms and model frameworks.

[0083] Suppose the configured computing resources are a first processor (e.g., CPU) and N second processors (e.g., GPUs), the dynamic scheduler can split the model to be trained into N parts and provide them to the N second processors.

[0084] For example, if the model to be trained has N x M network layers, the dynamic scheduler can take the first M network layers as the first part of the model to be trained, and the first pipeline corresponding to the first second processor can be used to train the first M network layers of the model to be trained; the M+1 to 2M network layers can be taken as the second part of the model to be trained, and the second pipeline corresponding to the second second processor can be used to train the M+1 to 2M network layers of the model to be trained; and so on, the (N-1) x M+1 to N x M network layers can be taken as the Nth part of the model to be trained, and the Nth pipeline corresponding to the Nth second processor can be used to train the (N-1) x M+1 to N x M network layers of the model to be trained.

[0085] It should be understood that the embodiments of the present disclosure do not specifically limit the number M of network layers each part of the model to be trained has, and can be set according to more practical application scenarios.

[0086] Among them, the second processors corresponding to adjacent pipelines can transmit data through a communication link. In this way, in the subsequent forward phase, the training data can be calculated through the first M network layers on the first second processor to obtain the forward calculation result of the first part, and the forward calculation result of the first part is transmitted to the second second processor, then the forward calculation result of the second part output by the M+1 to 2M network layers is calculated on the second second processor, and the forward calculation result of the second part is transmitted to the third second processor, and so on, until the forward calculation result of the (N-1) x M network layer output by the (N-1) part is transmitted to the Nth second processor on the (N-1) second processor, and the forward calculation result of the Nth part is obtained on the Nth second processor through the calculation of the N x M network layers.

[0087] It should be understood that the process of the backward phase is similar to that of the forward phase, and the network layers on each second processor will update the parameters using the gradients calculated by the backpropagation process.

[0088] Further, the first processor can transmit data to the second processors via the communication links, so as to assist the first or the Nth second processor to train the corresponding part of the to-be-trained model in the subsequent pipeline parallel training process of the to-be-trained model, so as to effectively eliminate the idle time caused by waiting for data or calculation dependency, thereby significantly improving the continuity and efficiency of the pipeline parallel training process.

[0089] In the pipeline parallel training process of the to-be-trained model by the N second processors, the dynamic scheduler can call the first processor to assist the first second processor to train the first part of the to-be-trained model, and assist the Nth second processor to train the Nth part of the to-be-trained model.

[0090] In a possible implementation, the scheduling of the first second processor and the first processor to jointly perform forward calculation on the first part allocated to the first second processor to obtain the forward calculation result of the first part includes: before the end of each round of distributed training of the to-be-trained model, scheduling the first processor to pre-read training data of the next round according to the first part, and pre-process the training data of the next round to obtain pre-fetch data of the next round; in response to starting the distributed training of the next round of the to-be-trained model, scheduling the first second processor to determine the forward calculation result of the first part according to the pre-fetch data from the first processor. The pre-processing of the first processor can include mapping operation of an embedding layer, dimension mapping, size transformation and the like.

[0091] For example, the first processor can start a data pre-fetch mechanism. For example, for the first round of training of the to-be-trained model, the first processor can asynchronously load the first batch of training data from the data source into the memory in advance, ready to input to the first second processor. For other rounds of training of the to-be-trained model, the first processor can be executed asynchronously with the first second processor, eliminating the delay caused by data loading. For example, for the Pth round of training of the to-be-trained model, while the first second processor is executing the (P-1) th round of training, the first processor can pre-read the training data of the Pth round and pre-process the training data of the Pth round, for example, including data conversion and other preprocessing operations, to obtain pre-fetch data of the Pth round; in response to starting the forward phase of the distributed training of the Pth round of the to-be-trained model, the first second processor can determine the forward calculation result of the first part output of the to-be-trained model according to the pre-fetch data of the Pth round from the first processor.

[0092] For example, the first processor can start a data pre-fetch mechanism. For example, for the first round of training of the to-be-trained model, the first processor can asynchronously load the first batch of training data from the data source into the memory in advance, ready to input to the first second processor. For other rounds of training of the to-be-trained model, the first processor can be executed asynchronously with the first second processor, eliminating the delay caused by data loading. For example, for the Pth round of training of the to-be-trained model, while the first second processor is executing the (P-1) th round of training, the first processor can pre-read the training data of the Pth round and pre-process the training data of the Pth round, for example, including data conversion and other preprocessing operations, to obtain pre-fetch data of the Pth round; in response to starting the forward phase of the distributed training of the Pth round of the to-be-trained model, the first second processor can determine the forward calculation result of the first part output of the to-be-trained model according to the pre-fetch data of the Pth round from the first processor. Figure 3As shown in the lower part, the first processor 1 trains the first part of the to-be-trained model in a cross manner of forward calculation and reverse calculation. When the second processor 1 is performing the reverse calculation of the first round of training, the first processor will simultaneously perform the pre-fetching and preparation of the data required for the forward calculation of the fifth round of training, so that the second processor 1 can obtain the pre-fetched data from the first processor as soon as the reverse calculation of the first round of training is completed, thereby reducing the time loss caused by data loading and preprocessing.

[0093] In this way, the data pre-fetching can be performed in the first stage of the pipeline parallel training, and the data required for the subsequent calculation is prepared, thereby eliminating the waiting time caused by data loading.

[0094] In this way, the data pre-fetching can be performed in the first stage of the pipeline parallel training, and the data required for the subsequent calculation is prepared, thereby eliminating the waiting time caused by data loading.

[0095] In a possible implementation, the scheduling of the Nth second processor and the first processor to jointly perform the forward calculation on the Nth part allocated to the Nth second processor to obtain the forward calculation result of the Nth part includes: scheduling the Nth second processor to perform the forward calculation on the Nth part to obtain the prediction result of the Nth part output; and scheduling the first processor to determine a loss value according to the prediction result and a preset loss function, and taking the loss value as the forward calculation result of the Nth part, where the loss value is used to guide the first processor and the N second processors to perform the reverse calculation.

[0096] For example, when the first second processor performs the forward calculation, the pre-fetched data obtained by the first processor can simultaneously satisfy the data required for the loss calculation (for example, the label information in the training data). Since the last part (the Nth part) of the to-be-trained model needs to perform additional loss calculation, the first processor can be scheduled to share this part of calculation, so that the Nth second processor can enter the reverse calculation from the forward calculation as soon as possible, and the waiting time caused by the loss calculation is eliminated.

[0097] In this way, the data pre-fetching can be performed in the first stage of the pipeline parallel training, and the data required for the subsequent calculation is prepared, thereby eliminating the waiting time caused by data loading.

[0098] Since the first part of the to-be-trained model contains additional data acquisition and mapping operations of the Embedding Layer, and the last part includes additional language model head (lm_head) calculation and loss calculation. If the first processor does not intervene in the backpropagation of the first part and the last part, the execution time of the first second processor and the Nth second processor will be greater than that of the other second processors in the backpropagation phase, considering the calculation dependency between the second processors, there will be idle waiting time.

[0099] When the first and last second processors alternately perform forward calculation and backpropagation, the first processor can intervene in the backpropagation to perform the backpropagation task that has met the data dependency. In order to reduce the idle waiting time in the model training process, the first processor can dynamically perform part of the backpropagation after processing the preprocessing task, such as calculation and transmission of gradients, and offload operation of the calculation subgraph, etc.

[0100] In a possible implementation, the first processor and the first second processor are scheduled to jointly perform backpropagation on the first part allocated to the first second processor, to obtain a backpropagation result of the first part, including: the first second processor is scheduled to perform backpropagation on the first M-K network layers in the first part, to obtain a first intermediate backpropagation result; and the first processor is scheduled to perform backpropagation on the last K network layers in the first part according to the first intermediate backpropagation result, to obtain the backpropagation result of the first part; where M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M. It should be understood that the value of K can be determined according to the actual application scenario, and the embodiments of the present disclosure do not make specific limitations on the specific value of K.

[0101] In the process of performing backpropagation on the first M-K network layers in the first part by the first second processor, the first processor can perform backpropagation on the last K network layers in parallel without data dependency, for example, the first processor can calculate the partial derivative of the parameter matrix in the K network layers, so as to obtain the backpropagation result of the first part as soon as possible after obtaining the first intermediate backpropagation result (for example, the gradient of the first M-K network layers in the first part).

[0102] The first processor assists the first second processor to jointly process the backpropagation of the first part of the to-be-trained model, and the backpropagation time of the first second processor can be balanced with the backpropagation time of the other second processors, so as to reduce the idle waiting time in the model training and improve the overall training efficiency of the to-be-trained model.

[0103] In a possible implementation, the scheduling the Nth second processor and the first processor to jointly perform back propagation on the Nth part to which the Nth second processor is assigned includes: scheduling the Nth second processor to perform back propagation on the first M-K network layers in the Nth part to obtain a second intermediate back propagation result; and scheduling the first processor to perform back propagation on the last K network layers in the Nth part according to the second intermediate back propagation result to obtain the back propagation result of the Nth part, where M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M. It should be understood that the value of K can be determined according to an actual application scenario, and embodiments of the present disclosure do not limit the specific value of K.

[0104] In the process of performing back propagation on the first M-K network layers in the Nth part by the Nth second processor, the first processor can perform back propagation on the last K network layers in parallel without data dependency, for example, the first processor can calculate the partial derivative of the parameter matrix in the K network layers, so as to obtain the back propagation result of the first part as soon as possible after obtaining the second intermediate back propagation result (for example, the gradient of the first M-K network layers in the Nth part).

[0105] Scheduling the first processor to assist the Nth second processor to jointly process back propagation of the Nth part of the to-be-trained model can balance the back propagation time of the Nth second processor with that of other second processors, so as to reduce idle waiting time in model training and improve the overall training efficiency of the to-be-trained model.

[0106] In a possible implementation, the dynamic scheduler obtains a first operation time of the first processor for training the first part of the to-be-trained model and a second operation time of the first second processor for training the first part of the to-be-trained model, and adjusts the number K of network layers in which the first processor performs back propagation on the first part of the to-be-trained model according to the first operation time and the second operation time.

[0107] For example, in multiple rounds of training, if the first operation time of the first processor for training the first part of the to-be-trained model becomes longer and the second operation time of the first second processor for training the first part of the to-be-trained model becomes shorter, the dynamic scheduler can reduce the number K of network layers in which the first processor shares in back propagation of the first part according to a preset value; if the first operation time of the first processor for training the first part of the to-be-trained model becomes shorter and the second operation time of the first second processor for training the first part of the to-be-trained model becomes longer, the dynamic scheduler can increase the number K of network layers in which the first processor shares in back propagation of the first part according to a preset value.

[0108] In this way, the computing resource allocation can be automatically adjusted according to the real-time computing performance, so that the first second processor performs the reverse calculation for the same time as the other second processors, which is beneficial to optimize the execution efficiency of the model training and reduce the idle waiting time.

[0109] In a possible implementation, the dynamic scheduler obtains a third operation time of the first processor for training the Nth part of the to-be-trained model and a fourth operation time of the Nth second processor for training the Nth part of the to-be-trained model, and adjusts the network layer number K of the first processor for performing the reverse calculation on the Nth part of the to-be-trained model according to the third operation time and the fourth operation time.

[0110] For example, in the multi-round training, if the third operation time of the first processor for training the Nth part of the to-be-trained model becomes longer and the fourth operation time of the Nth second processor for training the Nth part of the to-be-trained model becomes shorter, the dynamic scheduler can reduce the network layer number K of the first processor for sharing in the reverse calculation of the Nth part according to a preset value; if the third operation time of the first processor for training the Nth part of the to-be-trained model becomes shorter and the fourth operation time of the Nth second processor for training the Nth part of the to-be-trained model becomes longer, the dynamic scheduler can increase the network layer number K of the first processor for sharing in the reverse calculation of the Nth part according to a preset value.

[0111] In this way, the computing resource allocation can be automatically adjusted according to the real-time computing performance, so that the Nth second processor performs the reverse calculation for the same time as the other second processors, which is beneficial to optimize the execution efficiency of the model training and reduce the idle waiting time.

[0112] It can be known that, in this way of dynamic scheduling, the first processor and the second processor and other heterogeneous computing resources can work efficiently and cooperatively, and the resource utilization rate is maximized. This not only improves the overall efficiency of the training process, but also reduces the waste caused by the idle resources.

[0113] In a possible implementation, the method further includes: obtaining a second operation time of the first second processor for training the first part of the to-be-trained model, and a fifth operation time of any Qth processor of the second to (N-1)th second processors for training the Qth part of the to-be-trained model, where Q is any integer in the range of 2 to N-1; in a case where a difference between the second operation time and the fifth operation time is greater than a preset time threshold, it is indicated that the calculation time of the first second processor cannot be balanced with the processing time of the other second processors, and a first operation time of the first processor for training the first part of the to-be-trained model can be obtained; the network layer number K of the first processor for performing the first part of the reverse calculation task is adjusted according to the first time and the second time; or, in a case where the difference between the second operation time and the fifth operation time is less than or equal to the preset time threshold, the network layer number K of the first processor for performing the first part of the reverse calculation task can remain unchanged.

[0114] In a possible implementation, the method further includes: obtaining a fourth operation time of the Nth second processor for training the Nth part of the to-be-trained model, and a fifth operation time of any Qth processor of the second to (N-1)th second processors for training the Qth part of the to-be-trained model, where Q is any integer in the range of 2 to N-1; in a case where a difference between the fourth operation time and the fifth operation time is greater than a preset time threshold, it is indicated that the calculation time of the last second processor cannot be balanced with the processing time of the other second processors, and a third operation time of the first processor for training the Nth part of the to-be-trained model can be obtained; the network layer number K of the first processor for performing the Nth part of the reverse calculation task is adjusted according to the third time and the fourth time; or, in a case where the difference between the third operation time and the fifth operation time is less than or equal to the preset time threshold, the network layer number K of the first processor for performing the Nth part of the reverse calculation task can remain unchanged.

[0115] In a possible implementation, the method further includes: ending the training according to a preset number of training rounds or a performance of the to-be-trained model reaching a preset requirement. After the training is completed, the to-be-trained model can be saved in a preset storage device through a network, and a training report can be generated, which can include a performance indicator and a resource usage.

[0116] In summary, taking the to-be-trained model comprising the Transformer layer as an example, compared with the prior art method of reducing the Transformer layer at the beginning and end of the pipeline parallel training to reduce the idle waiting time, the model training method of the embodiment of the present disclosure can maintain the integrity of the to-be-trained model without reducing the Transformer layer, which is crucial for ensuring the performance and adaptability of the to-be-trained model, especially when a complex model needs to be trained to handle high-difficulty tasks. Maintaining the integrity of the Transformer layer means that it can be seamlessly integrated into the existing deep learning framework without major adjustments to the architecture of the to-be-trained model, which is easy to integrate and expand. This provides researchers and developers with a flexible and efficient parallel distributed training mode, which helps to accelerate the research and development and application promotion of deep learning models.

[0117] In addition, the model training method of the embodiment of the present disclosure uses the dynamic scheduler to schedule the first processor (e.g., CPU) to assist the first second processor (e.g., GPU) in training the first part of the to-be-trained model and the Nth second processor in training the Nth part of the to-be-trained model, which significantly improves the utilization of computing resources through the cooperative working mechanism of heterogeneous computing resources. In contrast, the prior art relies on the second processor (e.g., GPU) for computation, and does not fully utilize other resources such as the first processor (e.g., CPU). The participation of the first processor (e.g., CPU) in training can effectively reduce the waiting time of each second processor (e.g., GPU) and improve the overall training efficiency of the to-be-trained model.

[0118] In addition, the model training method of the embodiment of the present disclosure optimizes the allocation and utilization of computing resources, such as through data prefetching and completing part of the computing task on the first processor (e.g., CPU), which reduces the idle waiting time of the second processor during the training process, not only improving the efficiency of pipeline parallel training, but also making the training process smoother, reducing the waste of resources caused by waiting, and improving the cost-effectiveness of the training task. Especially in large-scale training tasks, it can significantly reduce the additional cost caused by uneven or idle computing resources, achieving higher economic benefits.

[0119] Therefore, the model training method of the embodiment of the present disclosure can reduce the idle waiting time in the training process, improve the overall training efficiency, eliminate the idle waiting time while maintaining the integrity of the structure of the to-be-trained model, and optimize the cooperative working mechanism of heterogeneous computing resources, significantly improving the efficiency of pipeline parallel distributed training.

[0120] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments without deviating from the principle logic. Limited by the length of the present disclosure, the present disclosure will not be described again. Those skilled in the art can understand that in the above-mentioned method of the specific embodiment, the specific execution order of each step should be determined according to its function and possible internal logic.

[0121] In addition, the present disclosure also provides a model training device, an electronic device, a computer readable storage medium, and a program, all of which can be used to implement any one of the model training methods provided by the present disclosure. The corresponding technical solutions and descriptions are described in the method section and are not repeated.

[0122] Figure 4 A block diagram of a model training device according to an embodiment of the present disclosure is shown as follows. Figure 4 As shown, the device is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors different from the first processor, and the device comprises:

[0123] The splitting module 41 is configured to split the to-be-trained model into N parts and assign the N parts to the N second processors respectively, wherein a pipeline corresponding to the i-th second processor is configured to train the i-th part of the to-be-trained model, each part of the to-be-trained model has the same number of network layers, and i is an integer within 1 to N.

[0124] The training module 42 is configured to cooperatively schedule the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model, and obtain a trained target model; wherein the first processor is configured to jointly train the first part of the to-be-trained model with the first second processor, and jointly train the Nth part of the to-be-trained model with the Nth second processor, and the time for the first second processor to train the first part of the to-be-trained model and the time for the Nth second processor to train the Nth part of the to-be-trained model are the same as the time for the remaining second processors to train the respective corresponding parts of the to-be-trained model.

[0125] In a possible implementation, the training module 42 is configured to: for the first second processor, schedule the first second processor and the first processor to jointly perform forward calculation and reverse calculation on the first part of interlaces allocated to the first second processor to obtain forward calculation results and reverse calculation results of the first part; for any processor in the second second processor to the N-1th second processor, schedule each ith processor to perform forward calculation and reverse calculation on the ith part of interlaces allocated to the ith processor to obtain forward calculation results and reverse calculation results of the ith part; for the Nth second processor, schedule the Nth second processor and the first processor to jointly perform forward calculation and reverse calculation on the Nth part of interlaces allocated to the Nth second processor to obtain forward calculation results and reverse calculation results of the Nth part; and obtain the trained target model according to N reverse calculation results corresponding to the N parts.

[0126] In a possible implementation, the scheduling the first second processor and the first processor to jointly perform forward calculation on the first part of interlaces allocated to the first second processor to obtain forward calculation results of the first part includes: before the end of each round of distributed training of the to-be-trained model, scheduling the first processor to pre-read training data of the next round according to the first part, and to pre-process the training data of the next round to obtain pre-fetch data of the next round; and in response to starting distributed training of the to-be-trained model in the next round, scheduling the first second processor to determine the forward calculation results of the first part according to the pre-fetch data from the first processor.

[0127] In a possible implementation, the scheduling the first second processor and the first processor to jointly perform reverse calculation on the first part of interlaces allocated to the first second processor to obtain reverse calculation results of the first part includes: scheduling the first second processor to perform reverse calculation on the first M-K network layers in the first part to obtain first intermediate reverse calculation results; and scheduling the first processor to perform reverse calculation on the last K network layers in the first part according to the first intermediate reverse calculation results to obtain the reverse calculation results of the first part; wherein M represents the number of network layers in each part of the to-be-trained model, and K is an integer less than M.

[0128] In a possible implementation, the apparatus further includes an adjusting module configured to: acquire a first operation time of the first processor for training the first part of the to-be-trained model, and a second operation time of a first second processor for training the first part of the to-be-trained model; and adjust a network layer number K of the first processor for performing back propagation in the first part of the to-be-trained model according to the first operation time and the second operation time.

[0129] In a possible implementation, the scheduling the Nth second processor and the first processor to jointly perform forward propagation on the Nth part allocated to the Nth second processor to obtain a forward propagation result of the Nth part includes: scheduling the Nth second processor to perform forward propagation on the Nth part to obtain a prediction result output by the Nth part; and scheduling the first processor to determine a loss value according to the prediction result and a preset loss function, and taking the loss value as the forward propagation result of the Nth part, where the loss value is used to guide the first processor and the N second processors to perform back propagation.

[0130] In a possible implementation, the scheduling the Nth second processor and the first processor to jointly perform back propagation on the Nth part allocated to the Nth second processor to obtain a back propagation result of the Nth part includes: scheduling the Nth second processor to perform back propagation on the first M-K network layers in the Nth part to obtain a second intermediate back propagation result; and scheduling the first processor to perform back propagation on the last K network layers in the Nth part according to the second intermediate back propagation result to obtain the back propagation result of the Nth part, where M represents a number of network layers in each part of the to-be-trained model, and K is an integer less than M.

[0131] In a possible implementation, the adjusting module is further configured to: acquire a third operation time of the first processor for training the Nth part of the to-be-trained model, and a fourth operation time of an Nth second processor for training the Nth part of the to-be-trained model; and adjust the network layer number K of the first processor for performing back propagation in the Nth part of the to-be-trained model according to the third operation time and the fourth operation time.

[0132] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0133] The embodiment of the present disclosure further provides a computer readable storage medium, having stored thereon computer program instructions, which, when executed by a processor, implement the method described above. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0134] The embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0135] The embodiment of the present disclosure further provides a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device executes the method described above.

[0136] The electronic device can be provided as a terminal, a server or other forms of devices.

[0137] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the method described above.

[0138] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a server operating system (Windows Server TM ), a graphical user interface operating system, a multi-user multi-process computer operating system (Unix TM ), a free and open source code Unix-like operating system (Linux TM ), an open source code Unix-like operating system (FreeBSD TM ) or the like.

[0139] In an example embodiment, a non-transitory computer-readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0140] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0141] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0142] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0143] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0144] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0145] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0146] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0147] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0148] The computer program product can be embodied in a tangible medium of

[0149] The above description of the various embodiments is intended to be illustrative in all aspects, rather than being restrictive. Those skilled in the art can refer to the description of the various embodiments to make modifications and / or improvements.

[0150] Those skilled in the art can understand that, in the above-described method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic.

[0151] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that it has entered the personal information collection range and will collect personal information. If the individual voluntarily enters the collection range, it is considered to agree to collect personal information. Or on the device for processing personal information, through the pop-up information or by asking the individual to upload his personal information, the individual's authorization is obtained under the condition that the device uses obvious mark / information to inform the individual of the personal information processing rules. The personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

[0152] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that, The method is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors different from the first processor, the method comprising: cutting a to-be-trained model into N parts and distributing the N parts to the N second processors respectively, wherein a pipeline of an i-th second processor is used to train an i-th part of the to-be-trained model, each part of the to-be-trained model has the same number of network layers, i is an integer within 1 to N; cooperatively scheduling the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model to obtain a trained target model; wherein the first processor is used to cooperatively train a first part of the to-be-trained model with a first second processor and train an Nth part of the to-be-trained model with an Nth second processor, and the first second processor trains the first part of the to-be-trained model for the same time as the Nth second processor trains the Nth part of the to-be-trained model, and the time for the rest of the second processors to train their respective corresponding parts of the to-be-trained model is the same.

2. The method of claim 1, wherein, cooperatively scheduling the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model to obtain a trained target model, comprising: for the first second processor, scheduling the first second processor and the first processor to cooperatively perform forward calculation and backward calculation on the first part allocated to the first second processor in an interleaved manner to obtain forward calculation results and backward calculation results of the first part; for any processor in the second second processor to the N-1th second processor, scheduling each i-th processor to perform forward calculation and backward calculation on the i-th part allocated to the i-th processor in an interleaved manner to obtain forward calculation results and backward calculation results of the i-th part; for the Nth second processor, scheduling the Nth second processor and the first processor to cooperatively perform forward calculation and backward calculation on the Nth part allocated to the Nth second processor in an interleaved manner to obtain forward calculation results and backward calculation results of the Nth part; obtaining a trained target model according to N backward calculation results corresponding to the N parts.

3. The method of claim 2, wherein, scheduling the first second processor and the first processor to cooperatively perform forward calculation on the first part allocated to the first second processor to obtain forward calculation results of the first part, comprising: before the end of each round of distributed training of the to-be-trained model, scheduling the first processor to pre-read training data of the next round according to the first part and pre-process the training data of the next round to obtain pre-fetch data of the next round; in response to starting the distributed training of the next round of the to-be-trained model, scheduling the first second processor to determine the forward calculation results of the first part according to the pre-fetch data from the first processor.

4. The method of claim 2, wherein, scheduling the first processor and the Nth second processor to jointly perform forward calculation on the Nth part to which the Nth second processor is assigned, to obtain a forward calculation result of the Nth part, comprising: scheduling the Nth second processor to perform forward calculation on the Nth part to obtain a prediction result output by the Nth part; scheduling the first processor to determine a loss value according to the prediction result and a preset loss function, and taking the loss value as the forward calculation result of the Nth part, wherein the loss value is used to guide the first processor and the N second processors to perform reverse calculation. scheduling the first processor and the Nth second processor to jointly perform reverse calculation on the Nth part to which the Nth second processor is assigned, to obtain a reverse calculation result of the Nth part, comprising:

5. The method of claim 4, wherein, scheduling the Nth second processor to perform reverse calculation on the first M-K network layers in the Nth part to obtain a second intermediate reverse calculation result; scheduling the first processor to perform reverse calculation on the last K network layers in the Nth part according to the second intermediate reverse calculation result, to obtain the reverse calculation result of the Nth part; wherein M represents the number of network layers that each part of the to-be-trained model has, and K is an integer less than M.

6. The method of claim 2, wherein, The method further comprises: obtaining a first operation time of the first processor training the first part of the to-be-trained model, and a second operation time of the first second processor training the first part of the to-be-trained model; adjusting the number K of network layers in which the first processor performs reverse calculation in the first part of the to-be-trained model according to the first operation time and the second operation time.

7. The method according to any one of claims 2 to 6, characterized in that, scheduling the first processor and the Nth second processor to jointly perform forward calculation on the Nth part to which the Nth second processor is assigned, to obtain a forward calculation result of the Nth part, comprising: scheduling the Nth second processor to perform forward calculation on the Nth part to obtain a prediction result output by the Nth part; scheduling the first processor to determine a loss value according to the prediction result and a preset loss function, and taking the loss value as the forward calculation result of the Nth part, wherein the loss value is used to guide the first processor and the N second processors to perform reverse calculation. scheduling the first processor and the Nth second processor to jointly perform reverse calculation on the Nth part to which the Nth second processor is assigned, to obtain a reverse calculation result of the Nth part, comprising:

8. The method of claim 7, wherein, scheduling the Nth second processor to perform reverse calculation on the first M-K network layers in the Nth part to obtain a second intermediate reverse calculation result; scheduling the first processor to perform reverse calculation on the last K network layers in the Nth part according to the second intermediate reverse calculation result, to obtain the reverse calculation result of the Nth part; wherein M represents the number of network layers that each part of the to-be-trained model has, and K is an integer less than M.

9. A model training apparatus characterized by comprising: The method further comprises: obtaining a first operation time of the first processor training the first part of the to-be-trained model, and a second operation time of the first second processor training the first part of the to-be-trained model; adjusting the number K of network layers in which the first processor performs reverse calculation in the first part of the to-be-trained model according to the first operation time and the second operation time. The device is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors different in type from the first processor, and the device comprises: The cutting module is configured to cut a to-be-trained model into N parts and distribute the N parts to the N second processors respectively, wherein an i-th second processor corresponds to a pipeline configured to train an i-th part of the to-be-trained model, each part of the to-be-trained model has the same number of network layers, i is an integer within 1 to N, and the to-be-trained model has the same number of network layers in each part; The training module is configured to cooperatively schedule the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model, and obtain a trained target model. The first processor is configured to train a first part of the to-be-trained model together with a first second processor and train an Nth part of the to-be-trained model together with an Nth second processor, and a time for the first second processor to train the first part of the to-be-trained model and a time for the Nth second processor to train the Nth part of the to-be-trained model are the same as a time for the remaining second processors to train respective corresponding parts of the to-be-trained model.

10. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 8. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 8. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 8. ​ 11. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​ 12. A computer program product comprising computer program instructions, characterised in that, ​

Images