Model training method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a model training method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computers, the method comprises the steps that a to-be-trained model is segmented into N parts to be distributed to N second processors respectively, and an assembly line corresponding to each second processor is used for training different parts of the to-be-trained model; cooperatively scheduling the first processor and the N second processors to perform pipeline parallel training on the to-be-trained model to obtain a trained target model; wherein the first processor is used for training the first part of the to-be-trained model together with the first second processor, and training the Nth part of the to-be-trained model together with the Nth second processor, so that the training time of each second processor for different parts of the to-be-trained model is the same. According to the embodiment of the invention, the idle waiting time of each second processor can be eliminated or reduced under the condition of keeping the integrity and compatibility of the to-be-trained model, and the model training efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method and apparatus, electronic device, storage medium and program product. Background Technology

[0002] With the rapid development of deep learning technology, especially in fields such as computer vision, natural language processing, and speech recognition, the scale and complexity of neural network models are growing exponentially. In the training of large-scale neural network models, pipelined parallelism is widely used to improve training efficiency. However, in pipelined parallel training, some computing devices (such as GPUs) experience idle time while waiting for upstream computations to complete, thus reducing overall training efficiency. Summary of the Invention

[0003] This disclosure presents a model training method and apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a model training method is provided, the method being applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors of different types from the first processor, the method comprising: dividing a model to be trained into N parts and assigning them to the N second processors respectively, wherein the pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained, each part of the model to be trained having the same number of network layers, i being an integer from 1 to N; co-scheduling the first processor and the N second processors to perform pipelined parallel training of the model to be trained to obtain a trained target model; wherein the first processor is used to jointly train the i-th part of the model to be trained with the first second processor and jointly train the N-th part of the model to be trained with the N-th second processor, and the time for the first second processor to train the i-th part of the model to be trained and the time for the N-th second processor to train the N-th part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

[0005] In one possible implementation, the first processor and the N second processors are collaboratively scheduled to perform pipelined parallel training on the model to be trained to obtain a trained target model. This includes: for the first second processor, scheduling the first second processor and the first processor to jointly perform forward and backward computations on the first part allocated to the first second processor in an alternating manner, to obtain the forward and backward computation results of the first part; for any processor from the second second processor to the (N-1)th second processor, scheduling each i-th processor to perform forward and backward computations on the i-th part allocated to it in an alternating manner, to obtain the forward and backward computation results of the i-th part; for the Nth second processor, scheduling the Nth second processor and the first processor to jointly perform forward and backward computations on the Nth part allocated to it in an alternating manner, to obtain the forward and backward computation results of the Nth part; and obtaining the trained target model based on the N backward computation results corresponding to the N parts.

[0006] In one possible implementation, scheduling the first second processor and the first processor to jointly perform forward computation on the first part allocated to the first second processor to obtain the forward computation result of the first part includes: before the end of each round of distributed training of the model to be trained, scheduling the first processor to pre-fetch the training data of the next round according to the first part, and preprocessing the training data of the next round to obtain the prefetched data of the next round; in response to the start of the next round of distributed training of the model to be trained, scheduling the first second processor to determine the forward computation result of the first part according to the prefetched data from the first processor.

[0007] In one possible implementation, scheduling the first second processor and the first processor to jointly perform reverse computation on the first part allocated to the first second processor to obtain the reverse computation result of the first part includes: scheduling the first second processor to perform reverse computation on the first MK network layers in the first part to obtain a first intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the first part based on the first intermediate reverse computation result to obtain the reverse computation result of the first part; wherein, M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0008] In one possible implementation, the method further includes: obtaining a first operation time for the first processor to train a first part of the model to be trained, and a second operation time for the first second processor to train a first part of the model to be trained; and adjusting the number of network layers K in which the first processor performs backpropagation in the first part of the model to be trained, based on the first operation time and the second operation time.

[0009] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform forward computation on the Nth part allocated to the Nth second processor to obtain the forward computation result of the Nth part includes: scheduling the Nth second processor to perform forward computation on the Nth part to obtain the prediction result output by the Nth part; scheduling the first processor to determine a loss value based on the prediction result and a preset loss function, and using the loss value as the forward computation result of the Nth part, wherein the loss value is used to guide the first processor and the Nth second processor to perform reverse computation.

[0010] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform reverse computation on the Nth part allocated to the Nth second processor to obtain the reverse computation result of the Nth part includes: scheduling the Nth second processor to perform reverse computation on the first MK network layers in the Nth part to obtain a second intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the Nth part based on the second intermediate reverse computation result to obtain the reverse computation result of the Nth part; wherein M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0011] In one possible implementation, the method further includes: obtaining a third operation time for the first processor to train the Nth part of the model to be trained, and a fourth operation time for the Nth second processor to train the Nth part of the model to be trained; and adjusting the number K of network layers K in which the first processor performs backpropagation in the Nth part of the model to be trained, based on the third operation time and the fourth operation time.

[0012] According to one aspect of this disclosure, a model training apparatus is provided, comprising: the apparatus applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors of a different type from the first processor; the apparatus comprising: a segmentation module for segmenting a model to be trained into N parts and distributing them to the N second processors respectively, wherein the pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained, each part of the model to be trained having the same number of network layers, i being an integer from 1 to N; and a training module for coordinating the first processor and the N second processors to perform pipelined parallel training of the model to be trained to obtain a trained target model; wherein the first processor is used to jointly train the first part of the model to be trained with the first second processor and jointly train the N-th part of the model to be trained with the N-th second processor, and the time taken by the first second processor to train the first part of the model to be trained and the time taken by the N-th second processor to train the N-th part of the model to be trained are the same as the time taken by the remaining second processors to train their respective corresponding parts of the model to be trained.

[0013] In one possible implementation, the training module is configured to: for a first second processor, schedule the first second processor and the first processor to perform forward and backward computations alternately on the first part allocated to the first second processor, to obtain the forward and backward computation results of the first part; for any processor from the second to the (N-1)th second processor, schedule each i-th processor to perform forward and backward computations alternately on the i-th part allocated to it, to obtain the forward and backward computation results of the i-th part; for the Nth second processor, schedule the Nth second processor and the first processor to perform forward and backward computations alternately on the N-th part allocated to it, to obtain the forward and backward computation results of the N-th part; and obtain the trained target model based on the N backward computation results corresponding to the N parts.

[0014] In one possible implementation, scheduling the first second processor and the first processor to jointly perform forward computation on the first part allocated to the first second processor to obtain the forward computation result of the first part includes: before the end of each round of distributed training of the model to be trained, scheduling the first processor to pre-fetch the training data of the next round according to the first part, and preprocessing the training data of the next round to obtain the prefetched data of the next round; in response to the start of the next round of distributed training of the model to be trained, scheduling the first second processor to determine the forward computation result of the first part according to the prefetched data from the first processor.

[0015] In one possible implementation, scheduling the first second processor and the first processor to jointly perform reverse computation on the first part allocated to the first second processor to obtain the reverse computation result of the first part includes: scheduling the first second processor to perform reverse computation on the first MK network layers in the first part to obtain a first intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the first part based on the first intermediate reverse computation result to obtain the reverse computation result of the first part; wherein, M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0016] In one possible implementation, the apparatus further includes an adjustment module for: obtaining a first operation time for the first processor to train a first part of the model to be trained, and a second operation time for the first second processor to train a first part of the model to be trained; and adjusting the number of network layers K in which the first processor performs backpropagation in the first part of the model to be trained, based on the first operation time and the second operation time.

[0017] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform forward computation on the Nth part allocated to the Nth second processor to obtain the forward computation result of the Nth part includes: scheduling the Nth second processor to perform forward computation on the Nth part to obtain the prediction result output by the Nth part; scheduling the first processor to determine a loss value based on the prediction result and a preset loss function, and using the loss value as the forward computation result of the Nth part, wherein the loss value is used to guide the first processor and the Nth second processor to perform reverse computation.

[0018] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform reverse computation on the Nth part allocated to the Nth second processor to obtain the reverse computation result of the Nth part includes: scheduling the Nth second processor to perform reverse computation on the first MK network layers in the Nth part to obtain a second intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the Nth part based on the second intermediate reverse computation result to obtain the reverse computation result of the Nth part; wherein M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0019] In one possible implementation, the adjustment module is further configured to: obtain a third operation time for the first processor to train the Nth part of the model to be trained, and a fourth operation time for the Nth second processor to train the Nth part of the model to be trained; and adjust the number K of network layers K in which the first processor performs backpropagation in the Nth part of the model to be trained, based on the third operation time and the fourth operation time.

[0020] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0021] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0022] According to one aspect of this disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the above-described method.

[0023] The model training method of this disclosure can be applied to a dynamic scheduler in a heterogeneous system. The heterogeneous system further includes a first processor and N second processors of different types from the first processor. The method includes: dividing the model to be trained into N parts and assigning them to the N second processors respectively, wherein the pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained, and each part of the model to be trained has the same number of network layers, where i is an integer from 1 to N; co-scheduling the first processor and the N second processors to perform pipelined parallel training on the model to be trained to obtain a trained target model; wherein the first processor is used to train the first part of the model to be trained together with the first second processor and to train the N-th part of the model to be trained together with the N-th second processor, and the time for the first second processor to train the first part of the model to be trained and the time for the N-th second processor to train the N-th part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

[0024] Therefore, this mechanism of working together through heterogeneous computing resources (such as the first processor and the second processor) optimizes the utilization of computing resources during pipelined parallel training. By executing some computing tasks on the first processor, the training time for each part of the model to be trained by each second processor is the same, thereby eliminating or reducing the idle waiting time caused by computational dependence during model training and improving the efficiency of model training.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0027] Figure 1 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram of a distributed training method according to an embodiment of the present disclosure is shown.

[0029] Figure 3 A schematic diagram showing a comparison of the effects of model training methods according to embodiments of the present disclosure is provided.

[0030] Figure 4A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown.

[0031] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0033] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0034] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0035] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0036] In related technologies, during pipelined parallel design, different computing devices (e.g., GPUs) may experience idle periods while waiting for dependent computations to complete, leading to wasted computing resources and reduced training efficiency, especially in large-scale neural network model training. To optimize computational efficiency in pipelined parallel processing, one network layer (e.g., a Transformer layer) can be removed from both ends of the pipeline to mitigate idle waiting time caused by computational dependencies, attempting to achieve a more compact pipelined parallel structure to reduce or eliminate idle waiting time.

[0037] This method in related technologies simplifies the computational task by directly reducing the number of network layers (e.g., Transformer layers) at both ends of the pipelined parallel training, thereby reducing waiting time and improving training efficiency. While simple to operate and directly reducing computational load, this method can, to some extent, reduce or eliminate idle waiting phenomena.

[0038] However, while reducing the number of network layers (e.g., Transformer layers) simplifies computation and reduces idle time, this approach sacrifices the integrity of the neural network model, potentially leading to performance degradation, especially compatibility issues when integrating with mainstream open-source model architectures. This is because most neural network models and frameworks are designed based on complete network layers (e.g., Transformer layers), and reducing the number of layers can cause the neural network model to perform worse than expected, or require significant adjustments to the neural network model architecture.

[0039] Therefore, although simplifying the number of layers in a neural network model is a viable alternative, its limitations in practical applications are obvious, especially in scenarios that pursue optimal model performance and maintain compatibility with deep learning frameworks. Reducing network layers (such as Transformer layers) will affect the performance of the neural network model and make it difficult to be compatible with mainstream open-source neural network model structures.

[0040] Furthermore, in related technologies, the utilization of computing resources during pipeline parallelism is uneven, primarily relying on high-performance computing devices such as GPUs, while utilizing other computing resources such as CPUs less. The CPU's participation in this process is extremely low, leading to underutilization and waste of computing resources. In view of this, embodiments of this disclosure provide a model training method that can eliminate or reduce idle waiting time, fully utilize computing resources, and improve model training efficiency while maintaining the integrity and compatibility of the model to be trained.

[0041] Figure 1 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the method is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further including a first processor and N second processors of different types from the first processor, the method comprising:

[0042] In step S11, the model to be trained is divided into N parts and assigned to the N second processors respectively. The pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained. Each part of the model to be trained has the same number of network layers, and i is an integer from 1 to N.

[0043] In step S12, the first processor and the N second processors are coordinated to perform pipelined parallel training on the model to be trained, so as to obtain the trained target model.

[0044] Wherein, the first processor is used to train the first part of the model to be trained together with the first second processor, and to train the Nth part of the model to be trained together with the Nth second processor, and the time for the first second processor to train the first part of the model to be trained and the time for the Nth second processor to train the Nth part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

[0045] In one possible implementation, the heterogeneous system is composed of processors with different architectures. The heterogeneous system is an electronic device, such as a server, user equipment (UE), mobile device, user terminal, terminal device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle device, wearable device, etc. The embodiments of this disclosure do not limit the specific form of the heterogeneous system.

[0046] In one possible implementation, the model training method can be executed by a dynamic scheduler. The dynamic scheduler can be a processor running preset software or preset program code in a heterogeneous system. The preset software or preset program is used to dynamically schedule other processors and can be implemented by a hardware co-description language, assembly language, high-level language (e.g., C, C++, etc.), or scripting language. Alternatively, the dynamic scheduler can also be a logic circuit embedded in a heterogeneous system. The embodiments of this disclosure do not limit the form of the dynamic scheduler.

[0047] In one possible implementation, the first processor in the heterogeneous system may include, but is not limited to, a central processing unit (CPU), a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a tensor processing unit (TPU), a field-programmable gate array (FPGA), etc.

[0048] In one possible implementation, the second processor in the heterogeneous system may include, but is not limited to, a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a neural network processing unit (NPU), etc.

[0049] It should be understood that the first processor and the second processor are of different types, and the embodiments of this disclosure do not limit the specific types of the first processor and the second processor.

[0050] In one possible implementation, the model training method can be designed for distributed training scenarios, where N second processors (e.g., multi-core GPUs, GPUs with multiple stream processor units) and second processors (e.g., CPUs) can be interconnected and communicate with each other. The model training method of this disclosure embodiment can effectively allocate training tasks to the first processor (e.g., CPU) and N second processors (e.g., GPUs), thereby improving the speed and efficiency of distributed training.

[0051] In one possible implementation, the model to be trained is a large artificial intelligence model, such as the Transformer model. The Transformer model is a deep learning architecture based on an attention mechanism, which may include multiple network layers (e.g., Transformer layers). Each Transformer layer may consist of an encoder and a decoder. The encoder is responsible for extracting features from the input sequence, and the decoder is responsible for generating the target sequence.

[0052] The training process of the model to be trained can include a forward phase and a backward phase. In the forward phase, training data is fed into the model in batches, and the model transforms the input training data into an output result, which is the model's prediction of the training data. Then, the model can call a loss function to calculate the loss value (e.g., error) between the output result and the true label. In the backward phase, the weight parameters of the model can be updated using the loss value obtained from the forward nodes, making the model's output result closer to the true label. The training process of the model to be trained can include multiple rounds of training, each round consisting of a forward phase and a backward phase. Through multiple rounds of iterative training, the model can gradually learn how to extract useful features from the input training data and produce more accurate prediction results until a well-trained target model is obtained.

[0053] In one possible implementation, for a training model with an extremely large number of parameters (e.g., trillions), a single-machine, single-card processor cannot meet the requirements for training such a large-scale model. Because the model is too large, it cannot be placed on a single processor. Therefore, in step S11, the dynamic scheduler can perform a splitting operation on the training model, allocating different network layers of the model to N second processors. This reduces the memory consumption of a single computing device, thereby enabling the training of the extremely large-scale model.

[0054] Figure 2 A schematic diagram of a distributed training method according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the first pipeline corresponding to the first second processor is used to train the first part of the model to be trained, the second pipeline corresponding to the second second processor is used to train the second part of the model to be trained, and so on, the Nth pipeline corresponding to the Nth second processor is used to train the Nth part of the model to be trained.

[0055] During each training round, the first part of the model to be trained will include additional data acquisition and embedding layer mapping operations, while the Nth part includes additional language model head (lm_head) computation and loss calculation. If the first processor does not participate in training, the execution time of the first and Nth second processors will be greater than the execution time of the other second processors. Considering the computational dependencies between the second processors, there will be idle waiting time.

[0056] Therefore, the computational tasks in the first and last pipeline stages need to be redesigned. In step S12, the dynamic scheduler can schedule the first processor to assist the first second processor in training the first part of the model to be trained, and to assist the Nth second processor in training the Nth part of the model to be trained. In this way, the dynamic scheduler can coordinate the scheduling of the first processor and N second processors to jointly perform pipelined parallel training of the model to be trained, and obtain the trained target model. This mechanism of working together through heterogeneous computing resources (such as the first processor and the second processor) optimizes the utilization of computing resources during pipelined parallel training and achieves an efficient training process. This mechanism may include, but is not limited to, strategies such as data prefetching and executing some computational tasks on the first processor to reduce the idle time of the second processor, so that the training time for each part of the model to be trained allocated to each second processor is the same.

[0057] It is evident that the dynamic scheduler can reduce resource waste and improve the overall system's computational efficiency by coordinating the scheduling of different types of computing resources (such as the first processor and the second processor). Furthermore, without altering the structure of the model to be trained (such as retaining the original Transformer layer of the model), the computational tasks of the first and last pipeline parallel stages can be redesigned to eliminate or reduce idle waiting time, achieving structural compatibility with various types of models to be trained and reducing the impact on the performance of the models to be trained.

[0058] In this way, by introducing the computation process involving the first processor at the beginning and end of the pipeline parallelism, the idle waiting time caused by waiting for data or computational dependencies can be effectively eliminated, thereby significantly improving the continuity and efficiency of the pipeline parallel training process.

[0059] In one possible implementation, step S12 may include: for the first second processor, scheduling the first second processor and the first processor to jointly perform forward and backward computations on the first part allocated to the first second processor in an alternating manner, to obtain the forward computation result and the backward computation result of the first part; for any processor from the second second processor to the (N-1)th second processor, scheduling each i-th processor to perform forward and backward computations on the i-th part allocated to it in an alternating manner, to obtain the forward computation result and the backward computation result of the i-th part; for the Nth second processor, scheduling the Nth second processor and the first processor to jointly perform forward and backward computations on the Nth part allocated to it in an alternating manner, to obtain the forward computation result and the backward computation result of the Nth part; and obtaining the trained target model based on the N backward computation results corresponding to the N parts.

[0060] For example, suppose there are 4 (N=4) second processors, namely: second processor 1 to second processor 4, where second processor 1 represents the first second processor, second processor 2 represents the second second processor, second processor 3 represents the third second processor, and second processor 4 represents the fourth second processor.

[0061] The dynamic scheduler can schedule the first processor and the second processor 1 to jointly train the first part of the model to be trained, schedule the second processor 2 to train the second part of the model to be trained, schedule the second processor 3 to train the third part of the model to be trained, and schedule the first processor and the second processor 4 to jointly train the fourth part of the model to be trained.

[0062] For each training round, the dynamic scheduler can schedule the first processor to assist the second processor 1 in performing forward computation, calculate the output of the first part of the model to be trained, and pass the output as the forward computation result 1 to the second processor 2.

[0063] The dynamic scheduler can schedule the second processor 2 to receive the forward computation result 1 output by the second processor 1, and perform forward computation based on the forward computation result 1 to calculate the output of the second part of the model to be trained, and pass the output as the forward computation result 2 to the second processor 3.

[0064] The dynamic scheduler can schedule the second processor 3 to receive the forward computation result 2 output by the second processor 2, and perform forward computation based on the forward computation result 2 to calculate the output of the third part of the model to be trained, and pass the output as the forward computation result 3 to the second processor 4.

[0065] The dynamic scheduler can schedule the second processor 4 to receive the forward computation result 3 output by the second processor 3, and schedule the first processor to assist the second processor 4 in performing forward computation together based on the forward computation result 3, calculate the output of the fourth part of the model to be trained, obtain the final model output of this round, calculate the loss value according to the loss function, and use the loss value as the forward computation result 4.

[0066] The dynamic scheduler can schedule the first processor to assist the second processor 4 in performing backpropagation based on the loss value, calculate the gradient 1 of the fourth part of the model to be trained, and use the gradient 1 to update the parameters of the fourth part of the model to be trained as the backpropagation result 1. Then, the gradient 1 and the loss value are passed back to the second processor 3.

[0067] The dynamic scheduler can schedule the second processor 3 to receive the gradient 1 and loss value output by the second processor 4, perform backpropagation based on the gradient 1 and loss value, calculate the gradient 2 of the third part of the model to be trained, and use the gradient 2 to update the parameters of the third part of the model to be trained as the backpropagation result 2, and then pass the gradient 2 and loss value to the second processor 2.

[0068] The dynamic scheduler can schedule the second processor 2 to receive the gradient 2 and loss value output by the second processor 3. The gradient 2 and loss value are used to perform backpropagation to calculate the gradient 3 of the second part of the model to be trained, and the parameters of the second part of the model to be trained are updated using the gradient 3 as the backpropagation result 3. Then the gradient 3 is passed to the second processor 1.

[0069] The dynamic scheduler can schedule the second processor 1 to receive the gradient 3 and loss value output by the second processor 2, and schedule the first processor to assist the second processor 1 in performing backpropagation together based on the gradient 3 and loss value to calculate the gradient 4 of the first part of the model to be trained, and use the gradient 4 to update the parameters of the first part of the model to be trained as the backpropagation result 4.

[0070] Since the reverse calculation results of each part of the model to be trained are also the parameter update results of each part, obtaining the reverse calculation results of each part completes the model training for this round.

[0071] To improve parallelism, each second processor can perform forward and backward computations interleaved on its assigned portion, where the forward and backward computations can be training tasks in different rounds.

[0072] If different second processors take different amounts of time to perform forward (or backward) computations on their respective assigned portions, each second processor may become idle while waiting for dependent computations to complete, reducing model training efficiency.

[0073] Therefore, considering that the training time for the first part and the last Nth part of the model to be trained is relatively long, for the first second processor, the first second processor and the first processor can be scheduled to perform forward and backward computations on the first part allocated to the first second processor in an alternating manner; for the Nth second processor, the Nth second processor and the first processor can be scheduled to perform forward and backward computations on the Nth part allocated to the Nth second processor in an alternating manner, so that the training time for each part of the model to be trained is the same for each second processor.

[0074] Figure 3 A schematic diagram showing a comparison of the effects of model training methods according to embodiments of the present disclosure is provided.

[0075] like Figure 3 As shown, second processor 1 is responsible for training the first part of the model to be trained, second processor 2 is responsible for training the second part, second processor 3 is responsible for training the third part, and second processor 4 is responsible for training the fourth part. The numbers "1" through "8" represent the training rounds; for example, "1" represents the first round, "2" represents the second round, and so on, with "8" representing the eighth round. Each training round includes forward computation and backward computation. In the forward computation of each round, second processor 2 depends on second processor 1, second processor 3 depends on second processor 2, and second processor 4 depends on second processor 3. In the backward computation of each round, second processor 3 depends on second processor 4, second processor 2 depends on second processor 3, and second processor 1 depends on second processor 2.

[0076] During the stable phase, each processor can perform forward and backward computations on a portion of the assigned training model in an alternating manner, where the forward and backward computations can be training tasks in different rounds. For example... Figure 3 As shown, during the stable phase, the second processor 1 can sequentially participate in the backward computation of the first round of training, the forward computation of the fifth round of training, the backward computation of the second round of training, the forward computation of the sixth round of training, the backward computation of the third round of training, the forward computation of the seventh round of training, the backward computation of the fourth round of training, the forward computation of the eighth round of training, and the backward computation of the fifth round of training.

[0077] Figure 3 The upper half shows the effect when the first processor is not involved in training. Because the second processor 1 takes a lot of time to train the first part of the model to be trained, and the second processor 4 takes a lot of time to train the last fourth part of the model to be trained, even in the stable phase, there will be a lot of idle waiting time caused by computational dependence.

[0078] Figure 3 The lower half shows the effect when the first processor participates in training. Since the first processor and the second processor 1 jointly train the first part of the model to be trained, and the second processor 4 jointly train the fourth part of the model to be trained, the time for the second processor 1 to train the first part of the model to be trained and the time for the second processor 4 to train the fourth part of the model to be trained are the same as the time for the other two second processors 2 and 3 to train their respective parts. In the stable phase, the idle waiting time caused by waiting for data or computational dependence can be effectively eliminated, thereby significantly improving the continuity and efficiency of the pipeline parallel training process.

[0079] It should be understood that Figure 3 This is merely an example of how each second processor performs forward and backward computations in an interleaved manner. The specific interleaving order can be set according to the actual application scenario, and the embodiments disclosed herein do not limit this.

[0080] In one possible implementation, distributed training of the model to be trained is completed to obtain a trained target model. The data to be processed is then input into the target model for processing to obtain the processing result. The target model is a model used to perform at least one task among image classification, object detection, image segmentation, and speech recognition. The data to be processed includes at least one of image data, speech data, and text data. The target model is trained according to the model training method described in steps S11-S12. This trained target model can be applied to various inference tasks such as image classification, object detection, image segmentation, and speech recognition, and can efficiently and accurately determine the inference result of the inference task.

[0081] The model training method of this disclosure embodiment will be described in detail below.

[0082] Before distributed training begins, the dynamic scheduler can perform initialization settings, such as configuring computing resources, initializing network connections, and loading training algorithms and model frameworks.

[0083] Assuming the configured computing resources are a first processor (e.g., CPU) and N second processors (e.g., GPU), the dynamic scheduler can divide the model to be trained into N parts and provide them to the N second processors.

[0084] For example, if the model to be trained has N×M network layers, the dynamic scheduler can use the first to M network layers as the first part of the model to be trained, and the first pipeline corresponding to the first second processor can be used to train the first to M network layers of the model to be trained; it can use the (M+1) to 2M network layers as the second part of the model to be trained, and the second pipeline corresponding to the second second processor can be used to train the (M+1) to 2M network layers of the model to be trained; and so on, the (N-1)×M+1 to N×M network layers can be used as the Nth part of the model to be trained, and the Nth pipeline corresponding to the Nth second processor can be used to train the (N-1)×M+1 to N×M network layers of the model to be trained.

[0085] It should be understood that the embodiments of this disclosure do not impose a specific limitation on the number M of network layers in each part of the model to be trained, and can be set for more practical application scenarios.

[0086] In this process, the second processors corresponding to adjacent pipelines can transmit data via a communication link. Thus, in the subsequent forward phase, the training data can be computed on the first second processor through the calculations of the first to M network layers to obtain the first part of the forward computation result. This first part of the forward computation result is then transmitted to the second second processor. The second second processor then computes the second part of the forward computation result output by the (M+1) to 2M network layers, and transmits this second part of the forward computation result to the third second processor, and so on, until the (N-1)th second processor transmits the (N-1)th part of the forward computation result output by the (N-1)×M network layer to the Nth second processor. Finally, the Nth second processor computes the Nth part of the forward computation result via the N×M network layer.

[0087] It should be understood that the backpropagation phase is similar to the forward phase, and the network layers on each second processor will use the gradients calculated during the backpropagation process to update their parameters.

[0088] Furthermore, the first processor can transmit data to the second processor via a communication link so that during the subsequent pipelined parallel training of the model to be trained, it can assist the first or Nth second processor in training the corresponding part of the model to be trained, thereby effectively eliminating idle time caused by waiting for data or computational dependencies, and thus significantly improving the continuity and efficiency of the pipelined parallel training process.

[0089] During the pipelined parallel training of the model to be trained by N second processors, the dynamic scheduler can call the first processor to assist the first second processor in training the first part of the model to be trained, and to assist the Nth second processor in training the Nth part of the model to be trained.

[0090] In one possible implementation, scheduling the first second processor and the first processor to jointly perform forward computation on the first part allocated to the first second processor to obtain the forward computation result of the first part includes: before the end of each round of distributed training of the model to be trained, scheduling the first processor to pre-fetch the training data of the next round according to the first part, and preprocessing the training data of the next round to obtain the prefetched data of the next round; in response to the start of the next round of distributed training of the model to be trained, scheduling the first second processor to determine the forward computation result of the first part according to the prefetched data from the first processor. The preprocessing of the first processor may include operations such as embedding layer mapping, dimension mapping, and size transformation.

[0091] For example, the first processor can initiate a data prefetching mechanism. For instance, for the first round of training of the model to be trained, the first processor can asynchronously load the first batch of training data from the data source into memory, ready for input to the first second processor. In subsequent rounds of training of the model to be trained, the first processor can execute asynchronously with the first second processor, eliminating delays caused by data loading. For example, for the Pth round of training of the model to be trained, when the first second processor executes the (P-1)th round of training, the first processor can prefetch the training data of the Pth round and preprocess the training data of the Pth round, including preprocessing operations such as data transformation, to obtain the prefetched data of the Pth round. Thus, in response to initiating the forward phase of distributed training of the model to be trained in the Pth round, the first second processor can determine the forward computation result of the first part of the output of the model to be trained based on the prefetched data of the Pth round from the first processor.

[0092] like Figure 3As shown in the lower part, the second processor 1 trains the first part of the model to be trained in a way that alternates between forward and backward computation. When the second processor 1 is performing the backward computation of the first round of training, the first processor will simultaneously perform the prefetching and preparation of the data required for the forward computation of the fifth round of training, so that the second processor 1 can obtain the prefetched data from the first processor as soon as it finishes the backward computation of the first round of training, thereby reducing the time loss caused by data loading and preprocessing.

[0093] This allows data prefetching to be performed in the first stage of pipelined parallel training, while simultaneously preparing the data needed for subsequent computations, thereby eliminating the waiting time caused by data loading.

[0094] In this process, the first second processor begins executing the first part of the pipelined parallel training's forward computation. At this time, the first processor can monitor data dependencies and computation progress, preparing for the next operation. When the first second processor begins executing the next round of forward computation, the first processor prepares the prefetched data to be used in the next round, including preprocessing operations such as data loading and data transformation.

[0095] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform forward computation on the Nth part allocated to the Nth second processor to obtain the forward computation result of the Nth part includes: scheduling the Nth second processor to perform forward computation on the Nth part to obtain the prediction result output by the Nth part; scheduling the first processor to determine a loss value based on the prediction result and a preset loss function, and using the loss value as the forward computation result of the Nth part, wherein the loss value is used to guide the first processor and the Nth second processor to perform reverse computation.

[0096] For example, when the first second processor performs forward computation, the prefetched data obtained by the first processor can simultaneously satisfy the data required for loss calculation (such as the annotation information in the training data). Since the last part (the Nth part) of the model to be trained requires additional loss calculation, the first processor can be scheduled to share the computation of this part, so that the Nth second processor can quickly move from forward computation to backward computation, eliminating the waiting time caused by the computation loss.

[0097] The first processor can input the prediction results and annotation information into a pre-loaded loss function, and use the loss function to calculate the loss value between the prediction results and the annotation information, so as to reduce the training time difference between the pipeline of the Nth second processor and the pipeline of other second processors.

[0098] Because the first part of the model to be trained includes additional data acquisition and embedding layer mapping operations, while the last part includes additional language model head (lm_head) computation and loss calculation, if the first processor does not participate in the reverse training of the first and last parts, the execution time of the first and Nth second processors will be greater than the execution time of the other second processors during the reverse phase. Considering the computational dependencies between the second processors, there will be idle waiting time.

[0099] While the first and last second processors alternately execute forward and backward computations, the first processor can intervene in the backward computation, performing backward computation tasks that satisfy the data dependencies. To reduce idle waiting time during model training, after completing the preprocessing tasks, the first processor can dynamically perform some backward computations based on the data dependencies, such as gradient calculation and propagation, and offload operations on the computation subgraph.

[0100] In one possible implementation, scheduling the first second processor and the first processor to jointly perform reverse computation on the first part allocated to the first second processor to obtain the reverse computation result of the first part includes: scheduling the first second processor to perform reverse computation on the first MK network layers in the first part to obtain a first intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the first part based on the first intermediate reverse computation result to obtain the reverse computation result of the first part; wherein, M represents the number of network layers in each part of the model to be trained, and K is an integer less than M. It should be understood that the value of K can be determined according to the actual application scenario, and the embodiments of this disclosure do not impose specific restrictions on the specific value of K.

[0101] In the process of the first second processor performing reverse computation on the first MK network layers in the first part, the first processor can perform reverse computation on the last K network layers in parallel without data dependency. For example, the first processor can calculate the partial derivatives of the parameter matrices in the K network layers so that after obtaining the first intermediate reverse computation result (e.g., the gradient of the first MK network layers in the first part), the reverse computation result of the first part can be obtained as soon as possible.

[0102] The first processor is scheduled to assist the first second processor in processing the reverse computation of the first part of the model to be trained. The reverse computation time of the first second processor can be balanced with the reverse computation time of other second processors to reduce idle waiting time in model training and improve the overall training efficiency of the model to be trained.

[0103] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform reverse computation on the Nth part allocated to the Nth second processor to obtain the reverse computation result of the Nth part includes: scheduling the Nth second processor to perform reverse computation on the first MK network layers in the Nth part to obtain a second intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the Nth part based on the second intermediate reverse computation result to obtain the reverse computation result of the Nth part; wherein, M represents the number of network layers in each part of the model to be trained, and K is an integer less than M. It should be understood that the value of K can be determined according to the actual application scenario, and the embodiments of this disclosure do not impose specific restrictions on the specific value of K.

[0104] In the process of the Nth second processor performing reverse computation on the first MK network layers in the Nth part, the first processor can perform reverse computation on the last K network layers in parallel without data dependency. For example, the first processor can calculate the partial derivatives of the parameter matrix in the K network layers so that after obtaining the second intermediate reverse computation result (e.g., the gradient of the first MK network layers in the Nth part), the reverse computation result of the first part can be obtained as soon as possible.

[0105] The first processor is scheduled to assist the Nth second processor in jointly processing the reverse computation of the Nth part of the model to be trained. The reverse computation time of the Nth second processor can be balanced with the reverse computation time of other second processors to reduce idle waiting time in model training and improve the overall training efficiency of the model to be trained.

[0106] In one possible implementation, the dynamic scheduler obtains the first operation time of the first processor training the first part of the model to be trained, and the second operation time of the first second processor training the first part of the model to be trained, and adjusts the number of network layers K in which the first processor performs backpropagation in the first part of the model to be trained based on the first operation time and the second operation time.

[0107] For example, in multi-round training, if the first operation time of the first processor training the first part of the model to be trained becomes longer, and the second operation time of the first second processor training the first part of the model to be trained becomes shorter, the dynamic scheduler can reduce the number of network layers K shared by the first processor in the reverse computation of the first part according to a preset value; if the first operation time of the first processor training the first part of the model to be trained becomes shorter, and the second operation time of the first second processor training the first part of the model to be trained becomes longer, the dynamic scheduler can increase the number of network layers K shared by the first processor in the reverse computation of the first part according to a preset value.

[0108] In this way, the allocation of computing resources can be automatically adjusted according to real-time computing performance, so that the time for the first second processor to perform reverse computation is the same as the time for other second processors to perform reverse computation. This is beneficial to optimizing the execution efficiency of model training and reducing idle waiting time.

[0109] In one possible implementation, the dynamic scheduler obtains the third operation time of the first processor training the Nth part of the model to be trained, and the fourth operation time of the Nth second processor training the Nth part of the model to be trained; based on the third operation time and the fourth operation time, the number K of network layers K in which the first processor performs backpropagation in the Nth part of the model to be trained is adjusted.

[0110] For example, in multi-round training, if the third operation time of the first processor training the Nth part of the model to be trained becomes longer, and the fourth operation time of the Nth second processor training the Nth part of the model to be trained becomes shorter, the dynamic scheduler can reduce the number of network layers K shared by the first processor in the reverse computation of the Nth part according to a preset value; if the third operation time of the first processor training the Nth part of the model to be trained becomes shorter, and the fourth operation time of the Nth second processor training the Nth part of the model to be trained becomes longer, the dynamic scheduler can increase the number of network layers K shared by the first processor in the reverse computation of the Nth part according to a preset value.

[0111] In this way, the allocation of computing resources can be automatically adjusted according to real-time computing performance, so that the time for the Nth second processor to perform reverse computation is the same as the time for other second processors to perform reverse computation. This is beneficial to optimizing the execution efficiency of model training and reducing idle waiting time.

[0112] This dynamically schedulable approach enables efficient collaboration between heterogeneous computing resources, such as the first and second processors, maximizing resource utilization. This not only improves the overall efficiency of the training process but also reduces waste caused by idle resources.

[0113] In one possible implementation, the method further includes: obtaining the second operation time of the first second processor training the first part of the model to be trained, and the fifth operation time of any Q-th processor among the second to N-1 second processors training the Q-th part of the model to be trained, where Q is any integer from 2 to N-1; if the difference between the second operation time and the fifth operation time is greater than a preset time threshold, it indicates that the computation time of the first second processor cannot be guaranteed to be balanced with the processing time of other second processors, and the first operation time of the first processor training the first part of the model to be trained can be obtained; adjusting the number of network layers K of the first processor executing the reverse computation task of the first part according to the first time and the second time; or, if the difference between the second operation time and the fifth operation time is less than or equal to the preset time threshold, the number of network layers K of the first processor executing the reverse computation task of the first part can remain unchanged.

[0114] In one possible implementation, the method further includes: obtaining the fourth operation time of the Nth second processor training the Nth part of the model to be trained, and the fifth operation time of any Qth processor among the 2nd to N-1th second processors training the Qth part of the model to be trained, where Q is any integer from 2 to N-1; if the difference between the fourth operation time and the fifth operation time is greater than a preset time threshold, it indicates that the computation time of the last second processor cannot be guaranteed to be balanced with the processing time of other second processors, and the third operation time of the first processor training the Nth part of the model to be trained can be obtained; adjusting the number of network layers K of the first processor executing the reverse computation task of the Nth part according to the third time and the fourth time; or, if the difference between the third operation time and the fifth operation time is less than or equal to the preset time threshold, the number of network layers K of the first processor executing the reverse computation task of the Nth part can remain unchanged.

[0115] In one possible implementation, the method further includes: ending training when the training reaches a preset number of training epochs or when the performance of the model to be trained meets preset requirements. After training is complete, the model to be trained can be saved to a preset storage device via a network, and a training report can be generated, which may include performance metrics and resource usage.

[0116] In summary, taking a trainable model containing Transformer layers as an example, compared to related technologies that require reducing Transformer layers at both ends of the pipelined parallel training to reduce idle waiting time, the model training method of this disclosure maintains the integrity of the trainable model without reducing Transformer layers. This is crucial for ensuring the performance and adaptability of the trainable model, especially when training complex models to handle challenging tasks. Maintaining the integrity of the Transformer layers means seamless integration into existing deep learning frameworks without significant adjustments to the architecture of the trainable model, facilitating integration and expansion. This provides researchers and developers with a flexible and efficient parallel distributed training mode, helping to accelerate the research and development and application of deep learning models.

[0117] Furthermore, the model training method of this disclosure, in which a dynamic scheduler schedules a first processor (e.g., CPU) to assist a first second processor (e.g., GPU) in training the first part of the model to be trained, and assists an Nth second processor in training the Nth part of the model to be trained, significantly improves the utilization rate of computing resources through a collaborative working mechanism of heterogeneous computing resources. In contrast, related technologies rely on second processors (e.g., GPUs) for computation, without fully utilizing other resources such as the first processor (e.g., CPU). The participation of the first processor (e.g., CPU) in training effectively reduces the waiting time of each second processor (e.g., GPU), improving the overall training efficiency of the model to be trained.

[0118] Furthermore, the model training method of this disclosure optimizes the allocation and utilization of computing resources. For example, by prefetching data and completing part of the computational tasks on a first processor (e.g., CPU), the idle waiting time of the second processor during training is reduced. This not only improves the efficiency of pipelined parallel training but also makes the training process smoother, reduces resource waste caused by waiting, and improves the cost-effectiveness of training tasks. Especially in large-scale training tasks, it can significantly reduce the additional costs caused by uneven or idle computing resources, achieving higher economic benefits.

[0119] Therefore, the model training method of this disclosure can reduce the idle waiting time during the training process, improve the overall training efficiency, eliminate the idle waiting time while maintaining the structural integrity of the model to be trained, and optimize the collaborative working mechanism of heterogeneous computing resources, significantly improving the efficiency of pipelined parallel distributed training.

[0120] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0121] In addition, this disclosure also provides a model training device, electronic device, computer-readable storage medium, and program, all of which can be used to implement any of the model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0122] Figure 4 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown, such as Figure 4 As shown, the device is applied to a dynamic scheduler in a heterogeneous system. The heterogeneous system further includes a first processor and N second processors of a different type from the first processor. The device includes:

[0123] The segmentation module 41 is used to segment the model to be trained into N parts and assign them to the N second processors respectively. The pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained. Each part of the model to be trained has the same number of network layers, and i is an integer from 1 to N.

[0124] Training module 42 is used to coordinate the first processor and the N second processors to perform pipelined parallel training on the model to be trained, so as to obtain a trained target model; wherein, the first processor is used to train the first part of the model to be trained together with the first second processor, and to train the Nth part of the model to be trained together with the Nth second processor, and the time for the first second processor to train the first part of the model to be trained and the time for the Nth second processor to train the Nth part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

[0125] In one possible implementation, the training module 42 is configured to: for the first second processor, schedule the first second processor and the first processor to perform forward and backward computations alternately on the first part allocated to the first second processor, to obtain the forward computation result and the backward computation result of the first part; for any processor from the second second processor to the (N-1)th second processor, schedule each i-th processor to perform forward and backward computations alternately on the i-th part allocated to it, to obtain the forward computation result and the backward computation result of the i-th part; for the Nth second processor, schedule the Nth second processor and the first processor to perform forward and backward computations alternately on the N-th part allocated to the Nth second processor, to obtain the forward computation result and the backward computation result of the N-th part; and obtain the trained target model based on the N backward computation results corresponding to the N parts.

[0126] In one possible implementation, scheduling the first second processor and the first processor to jointly perform forward computation on the first part allocated to the first second processor to obtain the forward computation result of the first part includes: before the end of each round of distributed training of the model to be trained, scheduling the first processor to pre-fetch the training data of the next round according to the first part, and preprocessing the training data of the next round to obtain the prefetched data of the next round; in response to the start of the next round of distributed training of the model to be trained, scheduling the first second processor to determine the forward computation result of the first part according to the prefetched data from the first processor.

[0127] In one possible implementation, scheduling the first second processor and the first processor to jointly perform reverse computation on the first part allocated to the first second processor to obtain the reverse computation result of the first part includes: scheduling the first second processor to perform reverse computation on the first MK network layers in the first part to obtain a first intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the first part based on the first intermediate reverse computation result to obtain the reverse computation result of the first part; wherein, M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0128] In one possible implementation, the apparatus further includes an adjustment module for: obtaining a first operation time for the first processor to train a first part of the model to be trained, and a second operation time for the first second processor to train a first part of the model to be trained; and adjusting the number of network layers K in which the first processor performs backpropagation in the first part of the model to be trained, based on the first operation time and the second operation time.

[0129] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform forward computation on the Nth part allocated to the Nth second processor to obtain the forward computation result of the Nth part includes: scheduling the Nth second processor to perform forward computation on the Nth part to obtain the prediction result output by the Nth part; scheduling the first processor to determine a loss value based on the prediction result and a preset loss function, and using the loss value as the forward computation result of the Nth part, wherein the loss value is used to guide the first processor and the Nth second processor to perform reverse computation.

[0130] In one possible implementation, scheduling the Nth second processor and the first processor to jointly perform reverse computation on the Nth part allocated to the Nth second processor to obtain the reverse computation result of the Nth part includes: scheduling the Nth second processor to perform reverse computation on the first MK network layers in the Nth part to obtain a second intermediate reverse computation result; scheduling the first processor to perform reverse computation on the last K network layers in the Nth part based on the second intermediate reverse computation result to obtain the reverse computation result of the Nth part; wherein M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

[0131] In one possible implementation, the adjustment module is further configured to: obtain a third operation time for the first processor to train the Nth part of the model to be trained, and a fourth operation time for the Nth second processor to train the Nth part of the model to be trained; and adjust the number K of network layers K in which the first processor performs backpropagation in the Nth part of the model to be trained, based on the third operation time and the fourth operation time.

[0132] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0133] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0134] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0135] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0136] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0137] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0138] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as a server operating system (Windows Server). TM Graphical user interface operating system, multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), an open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0139] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0140] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0141] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0142] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0143] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0144] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0149] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0150] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0151] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0152] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that, The method is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors of a different type from the first processor, the method comprising: The model to be trained is divided into N parts and assigned to the N second processors respectively. The pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained. Each part of the model to be trained has the same number of network layers, and i is an integer from 1 to N. The first processor and the N second processors are coordinated to perform pipelined parallel training on the model to be trained, so as to obtain a trained target model. Wherein, the first processor is used to train the first part of the model to be trained together with the first second processor, and to train the Nth part of the model to be trained together with the Nth second processor, and the time for the first second processor to train the first part of the model to be trained and the time for the Nth second processor to train the Nth part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

2. The method according to claim 1, characterized in that, The first processor and the N second processors are coordinated to perform pipelined parallel training on the model to be trained, resulting in a trained target model, including: For the first second processor, the first second processor and the first processor are scheduled to jointly perform forward and backward computations on the first part allocated to the first second processor in an alternating manner to obtain the forward computation result and the backward computation result of the first part; For any processor from the second to the (N-1)th second processor, schedule each i-th processor to perform forward and backward computations on the assigned i-th part in an alternating manner to obtain the forward and backward computation results of the i-th part; For the Nth second processor, the Nth second processor and the first processor are scheduled to jointly perform forward and backward computations on the Nth part allocated to the Nth second processor in an alternating manner to obtain the forward and backward computation results of the Nth part; Based on the N inverse calculation results corresponding to the N parts, the trained target model is obtained.

3. The method according to claim 2, characterized in that, The first second processor and the first processor are scheduled to jointly perform forward computation on the first part allocated to the first second processor, and the forward computation result of the first part is obtained, including: Before the distributed training of the model to be trained in each round ends, the first processor is scheduled to pre-fetch the training data of the next round according to the first part, and preprocess the training data of the next round to obtain the pre-fetched data of the next round. In response to the start of the next round of distributed training of the model to be trained, the first second processor is scheduled to determine the forward computation result of the first part based on the prefetched data from the first processor.

4. The method according to claim 2, characterized in that, The first second processor and the first processor are scheduled to jointly perform reverse computation on the first part allocated to the first second processor to obtain the reverse computation result of the first part, including: The first second processor is scheduled to perform reverse computation on the first MK network layers in the first part to obtain the first intermediate reverse computation result; The first processor is scheduled to perform reverse calculation on the last K network layers in the first part based on the first intermediate reverse calculation result, so as to obtain the reverse calculation result of the first part; Where M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

5. The method according to claim 4, characterized in that, The method further includes: The first operation time of the first processor training the first part of the model to be trained, and the second operation time of the first second processor training the first part of the model to be trained are obtained. Based on the first operation time and the second operation time, adjust the number of network layers K that the first processor performs inverse computation in the first part of the model to be trained.

6. The method according to claim 2, characterized in that, The Nth second processor and the first processor are scheduled to jointly perform forward computation on the Nth part allocated to the Nth second processor, and the forward computation result of the Nth part is obtained, including: The Nth second processor is scheduled to perform forward computation on the Nth part to obtain the prediction result output by the Nth part; The first processor is scheduled to determine a loss value based on the prediction result and a preset loss function, and the loss value is used as the forward calculation result of the Nth part, wherein the loss value is used to guide the first processor and the N second processors to perform reverse calculation.

7. The method according to any one of claims 2 to 6, characterized in that, The Nth second processor and the first processor are scheduled to jointly perform reverse computation on the Nth part allocated to the Nth second processor to obtain the reverse computation result of the Nth part, including: The Nth second processor is scheduled to perform reverse computation on the first MK network layers in the Nth part to obtain the second intermediate reverse computation result; The first processor is scheduled to perform reverse calculation on the last K network layers in the Nth part based on the second intermediate reverse calculation result, so as to obtain the reverse calculation result of the Nth part; Where M represents the number of network layers in each part of the model to be trained, and K is an integer less than M.

8. The method according to claim 7, characterized in that, The method further includes: The third operation time of the first processor training the Nth part of the model to be trained, and the fourth operation time of the Nth second processor training the Nth part of the model to be trained are obtained. Based on the third operation time and the fourth operation time, the number of network layers K in which the first processor performs backpropagation in the Nth part of the model to be trained is adjusted.

9. A model training device, characterized in that, The device is applied to a dynamic scheduler in a heterogeneous system, the heterogeneous system further comprising a first processor and N second processors of a different type from the first processor, the device comprising: The segmentation module is used to segment the model to be trained into N parts and assign them to the N second processors respectively. The pipeline corresponding to the i-th second processor is used to train the i-th part of the model to be trained. Each part of the model to be trained has the same number of network layers, and i is an integer from 1 to N. The training module is used to coordinate the scheduling of the first processor and the N second processors to perform pipelined parallel training on the model to be trained, so as to obtain the trained target model. Wherein, the first processor is used to train the first part of the model to be trained together with the first second processor, and to train the Nth part of the model to be trained together with the Nth second processor, and the time for the first second processor to train the first part of the model to be trained and the time for the Nth second processor to train the Nth part of the model to be trained are the same as the time for the other second processors to train their respective corresponding parts of the model to be trained.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images