Model training optimization method and model training system

Abstract

The present application provides a model training optimization method and a model training system. The method comprises: a main AI server determines R training devices, S batches, and the number W1 of batches that can be continuously executed in each training device, and on the basis of a forward computation duration of each forward task executed by each computing device, a backward computation duration of each backward task, communication durations of forward tasks or backward tasks corresponding to a same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, determines a first execution sequence of S forward tasks and S backward tasks on each training device, wherein the first execution sequence is an execution sequence under which a training duration of the R training devices using the S batches to train a neural network model satisfies a first condition, and the first execution sequence comprises at least one group of W2 continuously executed forward tasks. The first execution sequence determined by the method is conducive to improving the training efficiency of the neural network model.

Description

Model training optimization method and model training system

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese Patent Application No. 202410881521.6, filed on July 2, 2024, and entitled “A Model Training Optimization Method and Model Training System”, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to a model training optimization method and model training system. BACKGROUND

[0004] With the rapid development of artificial intelligence (AI) technology, neural network models are widely used in natural language processing, computer vision, speech recognition and other fields. In order to improve the understanding and reasoning ability of neural network models in various fields, the parameters in the neural network model show exponential growth, for example, the parameter amount of bidirectional encoder representations from transformer (BERT) model is 340 million, which brings great challenges to the training of neural network models. If more computing resources are used to train the neural network model, it will bring huge economic cost.

[0005] How to improve the training speed of the neural network model under the condition of limited computing resources is a technical problem that needs to be solved at present. SUMMARY

[0006] The present application provides a model training optimization method and model training system for improving the training speed of AI model under the condition of limited computing resources.

[0007] In a first aspect, the embodiments of the present application provide a model training optimization method, which can be executed by a main AI server. The method specifically includes the following steps: the main AI server determines R training devices, S batches and the number W1 of batches that can be continuously executed in each training device of a neural network model that performs pipeline parallel training, wherein R is less than S, and R, S and W1 are all greater than 0; the main AI server determines S forward tasks and S reverse tasks when training S batches on each training device, the forward calculation time of each forward task in the S forward tasks, the reverse calculation time of each reverse task in the S reverse tasks, and the communication time of the forward task or the reverse task corresponding to the same batch between two training devices; the main AI server determines the first execution order of the S forward tasks and the S reverse tasks on each training device in the R training devices according to the forward calculation time of each forward task, the reverse calculation time of each reverse task, the communication time of the forward task or the reverse task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, wherein the first execution order is an execution order in which the training time of the R training devices using the S batches to train the neural network model satisfies a first condition, and the first execution order includes at least one group of W2 continuous forward tasks, wherein W2 is greater than 0 and less than or equal to W1.

[0008] Through the above method, in the case of limited training devices, a first execution order for quickly training a neural network model can be determined. When the R training devices train the neural network model using the S batches according to the first execution order, the training time of the neural network model satisfies the first condition, so the determined first execution order can reduce the idle state of the training devices, which is conducive to improving the training efficiency of the neural network model.

[0009] In a possible design, the main AI server determines the first execution sequence of the S forward tasks and the S backward tasks on each of the R training devices according to the forward computation time length of each forward task executed by each computing device, the backward computation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, which can include: the main AI server determines an objective function and a constraint condition of the objective function according to the forward computation time length of each forward task executed by each computing device, the backward computation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device; the objective function is that the training time length of the neural network model satisfies a first condition; and the main AI server solves the objective function according to the constraint condition by using a linear solver, to obtain the first execution sequence of the S forward tasks and the S backward tasks on each of the R training devices.

[0010] In a possible design, the constraint condition includes at least one of the following:

[0011] the end moment of the s-th forward task on the r-th training device is less than the start moment of the s-th forward task on the r+1-th training device, r is an integer that traverses [1, R], and s is an integer that traverses [1, S]; or

[0012] the end moment of the s-th forward task on the r-th training device is less than or equal to the start moment of the s+1-th forward task on the r-th training device; or

[0013] the end moment of the s-th backward task on the r+1-th training device is less than the start moment of the s-th backward task on the r-th training device; or

[0014] the end moment of the s-th backward task on the r-th training device is less than or equal to the start moment of the s+1-th backward task on the r-th training device; or

[0015] the end moment of the s-th forward task on the r-th training device is less than or equal to the start moment of the s-th backward task on the r-th training device; or

[0016] in a case where the s1-th forward task is executed before the s2-th backward task, the end moment of the s1-th forward task on each training device is less than or equal to the start moment of the s2-th backward task; in a case where the s2-th backward task is executed before the s1-th forward task, the end moment of the s2-th backward task on each training device is less than or equal to the start moment of the s1-th forward task, s1 is an integer that traverses [1, S], s2 is an integer that traverses [1, S], and s1 is not equal to s2.

[0017] The number of batches successively executed in the rth training device is less than or equal to W1.

[0018] In a possible design, the first condition is that the training time length of the R training devices for training the neural network model by using the S batches in the first execution order is the smallest.

[0019] In a possible design, the method can further include: the master AI server establishing a forward communication domain for the forward tasks on the R training devices, and the master AI server establishing a backward communication domain for the backward tasks on the R training devices.

[0020] In a possible design, the master AI server establishing the forward communication domain for the forward tasks on the R training devices can include: performing the following steps on the rth training device of the R training devices: obtaining a first feature vector corresponding to an s-th forward task by performing the s-th forward task on the rth training device, the first feature vector being a forward calculation feature vector corresponding to a last network layer allocated to the rth training device; and establishing a forward communication domain between the rth training device and an (r+1)th training device; wherein the forward communication domain between the rth training device and the (r+1)th training device is used to send the first feature vector corresponding to the s-th forward task to the (r+1)th training device.

[0021] In a possible design, the master AI server establishing the backward communication domain for the backward tasks on the R training devices can include: performing the following steps on the rth training device of the R training devices: obtaining a second feature vector corresponding to an s-th backward task by performing the s-th backward task on the rth training device, the second feature vector being a backward calculation feature vector corresponding to a first network layer allocated to the rth training device; and establishing a backward communication domain between the rth training device and an (r-1)th training device; wherein the backward communication domain between the rth training device and the (r-1)th training device is used to send the second feature vector corresponding to the s-th backward task to the (r-1)th training device.

[0022] In a second aspect, the embodiments of the present application further provide a model training system, the model training system comprising a computing device and a training device,

[0023] The computing device is configured to: determine R training devices, S batches, and a number W1 of batches that can be continuously executed in each training device of a neural network model that is trained in a pipeline parallel manner, wherein R is less than S, and R, S, and W1 are greater than 0; determine S forward tasks and S backward tasks when training S batches on each training device, a forward calculation time length of each forward task in the S forward tasks, a backward calculation time length of each backward task in the S backward tasks, and a communication time length of a forward task or a backward task corresponding to a same batch of data between two training devices; determine, according to the forward calculation time length of each forward task executed by each computing device, the backward calculation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch of data between the two training devices, and the number W1 of batches that can be continuously executed in each training device, a first execution order of the S forward tasks and the S backward tasks on each training device in the R training devices, wherein the first execution order is an execution order in which the training time length of the R training devices in training the neural network model by using the S batches satisfies a first condition, and the first execution order includes at least one group of W2 continuous forward tasks, W2 is greater than 0 and less than or equal to W1.

[0024] The training device is configured to train the neural network model by using the S batches according to the first execution order.

[0025] By the above method, in the case that the training devices are limited, a first execution order for quickly training the neural network model can be determined, and when the R training devices train the neural network model by using the S batches according to the first execution order, the training time length of the neural network model satisfies the first condition, so that the determined first execution order can reduce the idle state of the training devices, and is beneficial to improve the training efficiency of the neural network model.

[0026] In a possible design, the computing device is specifically configured to: determine a target function and a constraint condition of the target function according to the forward calculation time length of each forward task executed by each computing device, the backward calculation time length of each backward task, the communication time length of the forward task or the backward task corresponding to a same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device; the target function is that the training time length of the neural network model satisfies the first condition; and a linear solver is used to solve the target function according to the constraint condition, to obtain the first execution order of the S forward tasks and the S backward tasks on each training device.

[0027] In a possible design, the computing device is specifically configured to: establish a forward communication domain for the forward tasks on the R training devices; and establish a backward communication domain for the backward tasks on the R training devices.

[0028] In a third aspect, the embodiments of the present application further provide a model training optimization device, which comprises a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program and execute the method provided in the first aspect.

[0029] In a fourth aspect, the embodiments of the present application further provide a model training optimization device, which comprises a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program and execute the method provided in the first aspect.

[0030] In a fifth aspect, the embodiments of the present application further provide a training device cluster comprising at least one training device, and each training device comprises a processor and a memory, wherein the processor of the at least one training device is configured to execute instructions stored in the memory of the at least one training device, so that the training device cluster executes the method provided in the first aspect.

[0031] In a sixth aspect, the embodiments of the present application further provide a computer program product comprising instructions, which, when executed by a training device cluster, cause the training device cluster to execute the method provided in the first aspect.

[0032] In a seventh aspect, the embodiments of the present application further provide a computer readable storage medium comprising computer program instructions, which, when executed by a training device cluster, cause the training device cluster to execute the method provided in the first aspect.

[0033] In an eighth aspect, the embodiments of the present application further provide a chip comprising instructions, which, when executed by a training device cluster, cause the training device cluster to execute the method provided in the first aspect.

[0034] In a ninth aspect, the embodiments of the present application provide a chip system comprising instructions, which, when executed by a training device cluster, cause the training device cluster to execute the method provided in the first aspect. The chip system can be composed of a chip, or can comprise a chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0035] FIG. 1 is a structural schematic diagram of a neural network model according to an embodiment of the present application;

[0036] FIG. 2 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0037] FIG. 3 is a structural schematic diagram of a system architecture according to an embodiment of the present application;

[0038] FIG. 4 is a structural schematic diagram of an AI server according to an embodiment of the present application;

[0039] FIG. 5 is a flowchart of a model training optimization method according to an embodiment of the present application;

[0040] FIG. 6 is a flowchart of a method for determining a CT(r, r+1, s, F)th forward communication duration according to an embodiment of the present application;

[0041] FIG. 7 is a flowchart of a method for determining a CT(r+1, r, s, B)th backward communication duration according to an embodiment of the present application;

[0042] FIG. 8 is a flowchart of a method for determining a first execution order according to an embodiment of the present application;

[0043] FIG. 9 is a flowchart of a method for determining a training duration of a neural network model according to an embodiment of the present application;

[0044] FIG. 10 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0045] FIG. 11 is a flowchart of a method for determining a t(1, S, B)th end time according to an embodiment of the present application;

[0046] FIG. 12 is a flowchart of a method for determining a t(r, s, B)th start time according to an embodiment of the present application;

[0047] FIG. 13 is a flowchart of a method for determining a t(r, s, B)th start time according to an embodiment of the present application;

[0048] FIG. 14 is a flowchart of a method for determining a t(r, s, B)th start time according to an embodiment of the present application;

[0049] FIG. 15 is a structural diagram of a cooling phase according to an embodiment of the present application;

[0050] FIG. 16 is a flowchart of a method for determining a t(r, s, B)th start time according to an embodiment of the present application;

[0051] FIG. 17 is a flowchart of a method for determining a t(r, x, F)th start time according to an embodiment of the present application;

[0052] FIG. 18 is a structural diagram of a stabilization phase according to an embodiment of the present application;

[0053] FIG. 19 is a flowchart of a method for determining a t(r, x, F)th start time according to an embodiment of the present application;

[0054] FIG. 20 is a structural diagram of a start-up phase according to an embodiment of the present application;

[0055] FIG. 21 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0056] FIG. 22 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0057] FIG. 23 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0058] FIG. 24 is a flowchart of a method for training a neural network model according to an embodiment of the present application;

[0059] FIG. 25 is a structural diagram of a model training optimization device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings. The specific operation methods, function descriptions, etc. in the method embodiments can also be applied to the device embodiments or system embodiments.

[0061] Hereinafter, some terms in the embodiments of the present application will be explained to facilitate understanding by those skilled in the art.

[0062] A neural network model is a complex network system formed by a large number of simple neurons widely connected with each other, which reflects many basic characteristics of brain functions and is a highly complex nonlinear dynamic learning system. The neural network model has large-scale parallelism, distributed storage and processing, self-organization, self-adaptation and self-learning ability, and is particularly suitable for processing information processing problems that need to consider many factors and conditions, and are imprecise and fuzzy.

[0063] Pipeline parallelism, which allocates different layers of a neural network model to different training devices for calculation, reduces the memory consumption of a single training device, thereby realizing the training of a large-scale neural network model. For example, as shown in FIG. 1, a neural network model includes three network layers, which are divided into three parts, wherein the first part includes the first network layer in the neural network model, the first part is allocated to a training device 1, the second part includes the second network layer in the neural network model, the second part is allocated to a training device 2, and the third part includes the third network layer in the neural network model, the third part is allocated to a training device 3.

[0064] One forward pass followed by one backward pass (1F1B) mode, a mode in which one forward pass and one backward pass are performed in cross.

[0065] Currently, when training a neural network model, a 1F1B mode pipeline parallel scheme is generally adopted to train the neural network model. The process of training the neural network model by the 1F1B mode pipeline parallel scheme will be described in detail in combination with specific embodiments.

[0066] It is assumed that the neural network model to be trained includes N network layers, the training set for training the neural network model includes M training data, and R training devices are used to train the neural network model, where N, M and R are positive integers. It should be understood that the training data can be text, image, voice, etc., which is not limited here. The training device can include an acceleration device, specifically, the acceleration device can be a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processing unit (DPU), etc., which is not limited here.

[0067] If the above neural network model including N network layers is trained by using the traditional 1F1B mode pipeline parallel scheme, a batch S to be divided from the M training data is obtained, S is a positive integer, and the M training data is evenly divided into S batches, and the number of training data in each batch is the same. It is assumed that the number of training data in each batch is m, where the number of training data in the training set M, the number of batches S, and the number of training data m in each batch satisfy the formula M = m * S.

[0068] Then, the N network layers are evenly distributed to the R training devices, and the number of network layers distributed to each training device is the same. It is assumed that the number of network layers distributed to each training device is n, where the number of network layers N in the neural network model, the number of training devices R, and the number of network layers n distributed to each training device satisfy the formula N = n * R.

[0069] After the division of the M training data and the distribution of the N network layers are completed, each training device in the R training devices combines the 1F1B mode pipeline parallel scheme to perform forward calculation and backward calculation on each batch in turn according to the distributed network layers, so as to achieve the goal of training the neural network model.

[0070] The pipeline parallelism scheme of the 1F1B mode is described in detail below in combination with an example shown in FIG. 2. The neural network model to be trained includes 100 network layers. The four training devices for training the neural network model are the first training device 1, the second training device, the third training device, and the fourth training device. Each training device is allocated 25 network layers. Specifically, the first 25 network layers of the neural network model are allocated to the first training device, the 26th to 50th network layers of the neural network model are allocated to the second training device, the 51st to 75th network layers of the neural network model are allocated to the third training device, and the 76th to 100th network layers of the neural network model are allocated to the fourth training device. The training set includes 1000 training data. If the 1000 training data are divided into five batches, each batch includes 200 training data. The time length for each training device to perform forward calculation on one training data according to one network layer of the neural network model is the same, and the time length for each training device to perform backward calculation on one training data according to one network layer of the neural network model is the same.

[0071] Each of the four training devices performs the following steps.

[0072] The first training device obtains the first batch, performs forward calculation on the 200 training data in the first batch according to the 25 network layers in the first training device, obtains a forward calculation feature vector corresponding to each of the 25 network layers in the first training device, takes the forward calculation feature vector corresponding to the 25th network layer of the neural network model as a first forward result corresponding to the first batch, and sends the first forward result corresponding to the first batch to the second training device.

[0073] After the second training device obtains the first forward result corresponding to the first batch, the second training device performs forward calculation on the first forward result corresponding to the first batch according to the 25 network layers in the second training device, obtains a forward calculation feature vector corresponding to each of the 25 network layers in the second training device, takes the forward calculation feature vector corresponding to the 50th network layer of the neural network model as a second forward result corresponding to the first batch, and sends the second forward result corresponding to the first batch to the third training device.

[0074] The first training device can obtain the second batch, perform forward calculation on the 200 pieces of training data in the second batch according to the 25 network layers in the first training device, obtain the forward calculation feature vector corresponding to each network layer of the 25 network layers in the first training device, take the forward calculation feature vector corresponding to the 25th network layer in the neural network model as the first forward result corresponding to the second batch, and send the first forward result corresponding to the second batch to the second training device.

[0075] After the third training device obtains the second forward result corresponding to the first batch, the third training device performs forward calculation on the second execution result corresponding to the first batch according to the 25 network layers in the third training device, obtains the forward calculation feature vector corresponding to each network layer of the 25 network layers in the third training device, takes the forward calculation feature vector corresponding to the 75th network layer in the neural network model as the third forward result corresponding to the first batch, and sends the third forward result corresponding to the first batch to the fourth training device.

[0076] While the third training device performs forward calculation on the second forward result corresponding to the first batch, the second training device can obtain the first forward result corresponding to the second batch, perform forward calculation on the first execution result corresponding to the second batch according to the 25 network layers in the second training device, obtain the forward calculation feature vector corresponding to each network layer of the 25 network layers in the second training device, take the forward calculation feature vector corresponding to the 50th network layer in the neural network model as the second forward result corresponding to the second batch, and send the second forward result corresponding to the second batch to the third training device.

[0077] While the third training device performs forward calculation on the second forward result corresponding to the first batch, the first training device can obtain the third batch, perform forward calculation on the 200 pieces of training data in the third batch according to the 25 network layers in the first training device, obtain the forward calculation feature vector corresponding to each network layer of the 25 network layers in the first training device, take the forward calculation feature vector corresponding to the 25th network layer in the neural network model as the first forward result corresponding to the third batch, and send the first forward result corresponding to the third batch to the second training device.

[0078] The fourth training device obtains the third forward result corresponding to the first batch, and performs forward calculation on the third forward result corresponding to the first batch according to the 25 network layers in the fourth training device to obtain a forward calculation feature vector corresponding to each network layer in the 25 network layers in the fourth training device, and takes the forward calculation feature vector corresponding to the 100th network layer in the neural network model as the fourth forward result corresponding to the first batch.

[0079] The fourth training device obtains the third forward result corresponding to the first batch, and performs forward calculation on the third forward result corresponding to the first batch according to the 25 network layers in the fourth training device to obtain a forward calculation feature vector corresponding to each network layer in the 25 network layers in the fourth training device, and takes the forward calculation feature vector corresponding to the 100th network layer in the neural network model as the fourth forward result corresponding to the first batch.

[0080] The rest is similar, which will not be repeated here.

[0081] In the above process, the four training devices can perform forward calculation and reverse calculation on the five batches respectively, wherein the start time of the model training is the start time of the first training device performing forward calculation on the first batch, and the end time of the model training is the end time of the first training device performing reverse calculation on the five batches. The training duration of the model training can be determined by the start time of the model training and the end time of the model training, that is, the training duration of the model training is equal to the end time of the model training minus the start time of the model training.

[0082] As can be seen from FIG. 2, in the process of training the neural network model by using the 1F1B mode pipeline parallel scheme, each training device can be in different stages at different times, which are the start-up stage, the stable stage and the cooling stage.

[0083] The start-up stage can be understood as the stage in which the training device continuously performs forward calculation. As can be seen from FIG. 2, the first training device continuously performs forward calculation on the first batch, the second batch, the third batch and the fourth batch. The time period in which the first training device continuously performs four forward calculations is the start-up stage corresponding to the first training device.

[0084] It should be understood that the determination method of the start-up stage corresponding to the second training device, the start-up stage corresponding to the third training device and the start-up stage corresponding to the fourth training device can refer to the determination method of the start-up stage corresponding to the first training device, which will not be repeated here.

[0085] The cooling stage can be understood as a stage in which the training devices continuously perform the back propagation. As shown in FIG. 2, the first training device continuously performs the back propagation on the third batch, the fourth batch and the fifth batch. The time period in which the first training device continuously performs the back propagation is the cooling stage corresponding to the first training device.

[0086] It should be understood that the cooling stage corresponding to the second training device, the cooling stage corresponding to the third training device and the cooling stage corresponding to the fourth training device can be determined according to the method of determining the cooling stage corresponding to the first training device, which will not be repeated here.

[0087] The stable stage can be understood as a stage in which the training devices alternately perform a forward propagation and a back propagation, or a stage other than the start stage and the cooling stage. As shown in FIG. 2, the first training device performs the back propagation on the first batch, the forward propagation on the fifth batch and the back propagation on the second batch. The time period in which the first training device alternately performs the forward propagation and the back propagation is the stable stage corresponding to the first training device.

[0088] It should be understood that the stable stage corresponding to the second training device, the stable stage corresponding to the third training device and the stable stage corresponding to the fourth training device can be determined according to the method of determining the stable stage corresponding to the first training device, which will not be repeated here.

[0089] As shown in the stable stage in FIG. 2, each training device alternately performs a forward propagation and a back propagation, that is, each training device performs the calculation in the 1F1B mode.

[0090] In addition, as shown in FIG. 2, there is a large amount of idle state of the training devices during the training of the neural network model. For example, when the fourth training device performs the back propagation on the fourth forward propagation result corresponding to the first batch, the first training device, the second training device and the third training device are all in the idle state, which will result in that the training devices cannot be effectively utilized, and thus the training time of the neural network model is long. Therefore, the embodiment of the present application provides a model training optimization method and device.

[0091] The embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0092] FIG. 3 shows a possible system architecture diagram to which the model training optimization method provided by the embodiment of the present application is applicable. The system architecture includes a terminal device 301 and an AI cluster 302, wherein the terminal device 301 and the AI cluster 302 can be connected through a wired or wireless manner.

[0093] The components of the system architecture will be described below.

[0094] (1) Terminal device 301

[0095] The user can input the number R of training devices for training the neural network model at the terminal device 301, and send the number R of training devices to the AI cluster 302 by the terminal device 301, and the AI cluster 302 can store the neural network model to be trained, wherein the neural network model includes N network layers, and the AI cluster 302 can determine the network layers in the neural network model allocated to each training device after receiving the number R of training devices. Alternatively,

[0096] The user can also input the number R of training devices for training the neural network model at the terminal device 301, and determine the number of network layers in the neural network model allocated to each training device at the terminal device 301, and then send the number R of training devices and the number of network layers in the neural network model allocated to each training device to the AI cluster 302 by the terminal device 301. The AI cluster 302 can store the neural network model to be trained, wherein the neural network model includes N network layers, and the AI cluster 302 can determine the network layers allocated to each training device according to the number of network layers allocated to each training device after receiving the number R of training devices and the number of network layers in the neural network model allocated to each training device.

[0097] It should be understood that the network layers allocated to each training device do not overlap, and the number of network layers allocated to each training device can be the same or different.

[0098] The user can input the number S of batches for dividing the training data at the terminal device 301, and send the number S of batches to the AI cluster 302 by the terminal device 301, and the AI cluster 302 can store the training set for training the neural network model, and the training set includes M training data, and the AI cluster 302 can determine the training data included in each batch after receiving the number S of batches. Alternatively,

[0099] The user can also input the number S of batches for dividing the training data at the terminal device 301, and determine the number of training data included in each batch at the terminal device 301, and then send the number S of batches and the number of training data included in each batch to the AI cluster 302 by the terminal device 301. The AI cluster 302 can store the training set for training the neural network model, and the training set includes M training data, and the AI cluster 302 can determine the training data included in each batch according to the number of training data included in each batch after receiving the number S of batches and the number of training data included in each batch.

[0100] It should be understood that the training data included in each batch does not overlap, and the number of training data included in each batch can be the same or different.

[0101] In addition, the terminal device 301 can provide the user with a first execution order of the forward task and the backward task corresponding to each batch of S batches on each training device, wherein the first execution order is generated and sent to the terminal device 301 by the AI cluster 302.

[0102] The terminal device is a device with wireless transceiver function, which can be deployed on land, including indoor or outdoor, handheld, wearable or vehicle-mounted; it can also be deployed on the water surface (such as ships, etc.); it can also be deployed in the air (such as airplanes, balloons and satellites, etc.). The terminal device can be a mobile phone, a tablet computer (Pad), a computer, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a vehicle-mounted terminal, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a wearable terminal, etc. The embodiments of the present application do not limit the application scenarios.

[0103] The terminal device 301 can be fixed or mobile. It can be understood that all or part of the functions of the terminal device 301 in the present application can also be realized by software functions running on hardware, or by virtualized functions instantiated on a platform (such as a cloud platform).

[0104] (2) AI cluster 302

[0105] The AI cluster 302 is composed of multiple AI servers, wherein each AI server at least includes a central processing unit (CPU) and an acceleration device, such as a GPU, a TPU, an NPU, a DPU, etc.

[0106] The following takes the structure of an AI server as an example, as shown in FIG. 4, including a CPU 401, a GPU 402, a communication interface 403, a high bandwidth memory (HBM) 404, and a bus 405. The CPU 401, the GPU 402, the communication interface 403, and the high bandwidth memory 404 communicate through the bus 405. It should be understood that the number of CPUs 401 and GPUs 402 is not limited by the present application.

[0107] The CPU 401 can determine the network layer in the neural network model allocated to each of the R training devices, the training data included in each of the S batches, and the number W1 of batches that can be continuously executed in each of the training devices, W1 being greater than 0, and determine the first execution order of the forward task and the backward task corresponding to the S batches on each of the R training devices according to the above information. The GPU 402 can train the neural network model according to the first execution order determined by the CPU 401.

[0108] The HBM 404 can be used to store program code for determining the first execution order, and can also be used to store temporary data generated by the CPU 401 in the process of determining the first execution order, and can also be used to store temporary data generated by the GPU 402 in the process of training the neural network model.

[0109] The bus 405 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 4, but it does not mean that there is only one bus or one type of bus. The bus 405 can include a path for transmitting information between the components of the AI server (e.g., the CPU 401, the GPU 402, the communication interface 403, and the HBM 404).

[0110] The communication interface 403 uses, for example but not limited to, a network interface card, a transceiver, etc., to realize the communication between the AI server and other devices or communication networks.

[0111] There is at least one master AI server in a plurality of AI servers, and other AI servers except the master AI server can be regarded as training devices for training the neural network model.

[0112] The main AI server stores the neural network model to be trained and the training set. The CPU in the main AI server determines the network layer in the neural network model allocated to each of the R training devices, the training data included in each of the S batches, and the number W1 of batches that can be continuously executed in each training device. After that, the first execution order of the S forward tasks and the S backward tasks on each of the R training devices can be determined according to the above information, wherein the first execution order is an execution order in which the training time of the R training devices using the S batches to train the neural network model satisfies a first condition. The first execution order includes at least one group of W2 continuous forward tasks, and W2 is less than or equal to W1. Then, the GPU in each of the R training devices trains the neural network model using the S batches according to the first execution order.

[0113] The main AI server in the AI cluster 302 can also provide the following capabilities, such as model building and training, model management, model deployment, and model optimization and update. The various capabilities provided by the main AI server can be integrated for user use, or independent functions can be provided to the user.

[0114] In combination with the system architecture diagram shown in FIG. 3, the present embodiment provides a model training optimization method, which can be executed by the main AI server shown in FIG. 3. Referring to FIG. 5, the flow of the method includes:

[0115] S501, the main AI server determines R training devices, S batches, and the number W1 of batches that can be continuously executed in each training device for pipeline parallel training of a neural network model, wherein R is less than S, and W1 is greater than 0.

[0116] In the present embodiment, the AI cluster in FIG. 3 includes a main AI server and other AI servers for training a neural network model, wherein the models of the AI servers in the AI cluster can be the same or different. In order to facilitate the training of the neural network model, generally, the main AI server determines the AI servers with the same model from the AI cluster as training devices. The main AI server can be a training device or not, which is not limited here.

[0117] In addition, the storage capacity of each training device determined by the main AI server can be the same or different, which is not limited here. It should be understood that the storage capacity of the training device can also be understood as the memory capacity or the display memory capacity of the training device.

[0118] In the embodiments of the present application, the neural network model to be trained can be an existing neural network model stored by the main AI server itself or input by the user to the main AI server through the terminal device. The neural network model includes N network layers, each of which can be any one of a convolutional layer, a pooling layer, an activation function layer, a fully connected layer, etc., or an attention structure, without limitation.

[0119] In the embodiments of the present application, after determining the R training devices for training the neural network model, the main AI server can determine the network layers in the neural network model allocated to each of the R training devices through two possible implementation manners, including implementation manner A1 and implementation manner A2.

[0120] In implementation manner A1, the user inputs the number R of training devices to the main AI server through the terminal device. Correspondingly, the main AI server receives the number R of training devices and selects R AI servers from the AI cluster as training devices, and the selected R training devices are of the same model. Then, the main AI server allocates the network layers in the neural network model to the R training devices. The network layers allocated to each training device are non-overlapping, and the number of network layers allocated to each training device can be the same or different.

[0121] In implementation manner A2, the user inputs the number R of training devices and the number of network layers in the neural network model allocated to each training device to the main AI server through the terminal device. Correspondingly, the main AI server receives the number R of training devices and the number of network layers in the neural network model allocated to each training device, and selects R AI servers from the AI cluster as training devices, and the selected R training devices are of the same model. Then, the main AI server allocates the network layers in the neural network model to the R training devices according to the number of network layers allocated to each training device. The network layers allocated to each training device are non-overlapping, and the number of network layers allocated to each training device can be the same or different.

[0122] In the above implementation manner A1 or implementation manner A2, the rth training device is allocated N(r) network layers, and the N(r) network layers allocated to the rth training device are part of the N network layers, where N(r) represents the number of network layers allocated to the rth training device, r is an integer that takes values in [1, R], and N(r) is an integer less than or equal to N.

[0123] It should be understood that the sum of the number of network layers allocated to the R training devices is N, i.e., N(1)+N(2)+…+N(R)=N, where N(1) represents the number of network layers allocated to the first training device, N(2) represents the number of network layers allocated to the second training device, and N(R) represents the number of network layers allocated to the Rth training device.

[0124] In the embodiments of the present application, the training set for training the neural network model can be uploaded by the user to the main AI server through the terminal device, or can be downloaded from the data platform by the main AI server, which is not limited herein. The training set can be stored in the main AI server, or can be stored in a database accessible by the main AI server, which is not limited herein.

[0125] For the convenience of the following description, in the embodiments of the present application, it is assumed that the training set includes M training data, and the data form of the training data can be text, image, voice, etc., which is not limited herein. It should be understood that the data form of the training data in the same training set is the same.

[0126] In the embodiments of the present application, after determining the S batches, the main AI server can determine the training data in the training set included in each batch in the S batches through the following two possible implementation manners, including implementation manner B1 and implementation manner B2.

[0127] In implementation manner B1, the user inputs the number S of batches to the main AI server through the terminal device; correspondingly, the main AI server receives the number S of batches, and allocates the M training data in the training set to the S batches. Wherein, the training data included in each batch is non-overlapping, and the number of training data included in each batch can be the same or different.

[0128] In implementation manner B2, the user inputs the number S of batches and the number of training data included in each batch to the main AI server through the terminal device; correspondingly, the main AI server receives the number S of batches and the number of training data included in each batch, and allocates the training data in the training set to the S batches according to the number of training data included in each batch. Wherein, the training data included in each batch is non-overlapping, and the number of training data included in each batch can be the same or different.

[0129] In the above implementation manner B1 or implementation manner B2, the s-th batch includes M(s) training data, and the M(s) training data included in the s-th batch is part of the M training data, where M(s) represents the number of training data included in the s-th batch, s is an integer taking values in [1, S], and M(s) is an integer less than or equal to M.

[0130] It should be understood that the sum of the number of training data included in the S batches is M, that is, M(1)+M(2)+…+M(S)=M, where M(1) represents the number of training data included in the first batch, M(2) represents the number of training data included in the second batch, and M(S) represents the number of training data included in the Sth batch.

[0131] In the embodiment of the present application, after determining the network layers in the neural network model assigned to each of the R training devices and the training data in the training set included in each of the S batches, the main AI server can determine the number W1 of batches that can be continuously executed in each of the training devices by the following method.

[0132] Specifically, the number W1 of batches that can be continuously executed in the rth training device of the R training devices is determined according to at least one of the following: the storage usage of the forward calculation feature vectors corresponding to the N(r) network layers, the storage usage of the backward calculation feature vectors corresponding to the N(r) network layers, and the storage capacity of the rth training device. Wherein the storage usage of the forward calculation feature vectors corresponding to the N(r) network layers is determined according to the N(r) network layers, and the storage usage of the backward calculation feature vectors corresponding to the N(r) network layers is determined according to the N(r) network layers.

[0133] In the embodiment of the present application, the number W1 of batches that can be continuously executed in the rth training device can be determined by the following two possible implementation manners:

[0134] In one possible implementation manner, the number W1 of batches that can be continuously executed in the rth training device can be determined by the following implementation manner: determining the storage usage corresponding to the forward calculation feature vectors corresponding to the N(r) network layers, and taking the ratio of the storage capacity of the rth training device to the storage usage of the forward calculation feature vectors corresponding to the N(r) network layers as the number W1 of batches that can be continuously executed in the rth training device.

[0135] In another possible implementation manner, the number W1 of batches that can be continuously executed in the rth training device can be determined by the following implementation manner: determining the sum of the storage usage corresponding to the forward calculation feature vectors corresponding to the N(r) network layers and the storage usage corresponding to the backward calculation feature vectors corresponding to the N(r) network layers as a first reference storage usage, and taking the ratio of the storage capacity of the rth training device to the first reference storage usage as the number W1 of batches that can be continuously executed in the rth training device.

[0136] S502, the main AI server determines S forward tasks and S reverse tasks when training S batches on each training device, a forward calculation time length of each forward task in the S forward tasks, a reverse calculation time length of each reverse task in the S reverse tasks, and a communication time length of a forward task or a reverse task corresponding to the same batch between two training devices.

[0137] In the embodiment of the application, the S batches on each training device correspond to one forward task and one reverse task respectively, that is, there are S forward tasks and S reverse tasks on each training device.

[0138] In the embodiment of the application, the main AI server can determine the forward calculation time length of each forward task in the S forward tasks on each training device by the following method:

[0139] Taking the T(r,s,F)th forward calculation time length as an example, where r in T(r,s,F) represents the rth training device, s represents the sth batch, and F represents forward calculation, the T(r,s,F)th forward calculation time length can represent the time length of performing forward calculation on the sth batch by the rth training device, or the T(r,s,F)th forward calculation time length can also represent the forward calculation time length of the sth forward task on the rth training device, r is an integer taking values from [1, R], and s is an integer taking values from [1, S].

[0140] In the embodiment of the application, the T(r,s,F)th forward calculation time length is determined according to the number N(r) of network layers in the rth training device, the number M(s) of training data in the sth batch, and the calculation performance of the rth training device. The calculation performance of the training device is determined by the model of the training device.

[0141] In a possible implementation, the main AI server can determine the T(r,s,F)th forward calculation time length by the following method: before performing the model training optimization method shown in FIG. 5, the main AI server can directly measure the unit time T F (r,1,F) of performing forward calculation on one training data by one network layer in the rth training device.

[0142] After determining the unit time T F (r,1,F), the T(r,s,F)th forward calculation time length is equal to the product of the unit time T F (r,1,F), N(r), and M(s), that is, the T(r,s,F)th forward calculation time length = T F (r,1,F)*N(r)*M(s), so the T(r,s,F)th forward calculation time length can be determined.

[0143] It should be understood that the determined unit time T F is not the same. In most cases, the R training devices determined by the main AI server are of the same model, that is, the computing performance of the R training devices is the same, so the main AI server can measure the unit time T F of each training device, which is the unit time of each training device performing forward calculation on one training data. F Without separately calculating the unit time T F of each training device performing forward calculation on one training data.

[0144] In the embodiments of the present application, the main AI server can determine the reverse calculation time of each reverse task on each training device S by the following method:

[0145] Taking the T(r,s,B)th reverse calculation time as an example, where r in T(r,s,B) represents the rth training device, s represents the sth batch, and B represents reverse calculation. The T(r,s,B)th reverse calculation time can represent the time of the rth training device performing reverse calculation on the sth batch, or the T(r,s,B)th reverse calculation time can also represent the reverse calculation time of the sth reverse task on the rth training device. r is an integer that takes [1,R], and s is an integer that takes [1,S].

[0146] In the embodiments of the present application, the T(r,s,B)th reverse calculation time is determined according to the number of N(r) network layers in the rth training device, the number of M(s) training data in the sth batch, and the computing performance of the rth training device. Wherein the computing performance of the training device is determined by the training device model.

[0147] In one possible implementation, the main AI server can determine the T(r,s,B)th reverse calculation time by the following method: Before the main AI server performs the model training optimization method shown in FIG. 5 in the present application, the main AI server can directly measure the unit time T B of each training device performing reverse calculation on one training data. It should be understood that the measurement method can be that the main AI server actually calls one network layer in the rth training device to perform reverse calculation on one training data.

[0148] After determining the unit time T B , the T(r,s,B)th reverse calculation time is equal to the unit time T B, the product of N(r) and M(s), i.e., the T(th) back propagation time length T B * N(r) * M(s), i.e., the T(th) back propagation time length T

[0149] It should be understood that the determined unit time T B is not the same. In most cases, the R training devices determined by the main AI server are of the same model, i.e., the computing performance of the R training devices is the same, and therefore, the main AI server can measure the unit time T B of each training device, which is the unit time T B of each training device, which is the unit time T B .

[0150] In the embodiments of the present application, the main AI server can determine the communication time length of the forward task or the back task corresponding to the same batch between two training devices by the following method, wherein the communication time length of the forward task corresponding to the same batch between two devices can be understood as the forward communication time length between two devices, and the communication time length of the back task corresponding to the same batch between two devices can be understood as the back communication time length between two devices:

[0151] Taking the CT(r, r+1, s, F)th forward communication time length as an example, wherein r in CT(r, r+1, s, F) represents the rth training device, r+1 represents the r+1th training device, s represents the sth batch, F represents forward calculation, r is an integer taking all values in [1, R], and s is an integer taking all values in [1, S]. The CT(r, r+1, s, F)th forward communication time length can represent the forward communication time length of the forward task corresponding to the sth batch between the rth device and the r+1th device, or the forward communication time length of the sth forward task between the rth device and the r+1th device, or the communication time length of the rth training device transmitting the first feature vector corresponding to the sth batch to the r+1th training device, wherein the first feature vector is the forward calculation feature vector corresponding to the last network layer in the N(r) network layers allocated to the rth training device, and the N(r) network layers are located in the N network layers, and N(r) is an integer less than or equal to N.

[0152] The main AI server can determine the CT(r, r+1, s, F)th forward communication time length by the following two possible implementation manners, including implementation manner C1 and implementation manner C2.

[0153] Embodiment C1 includes the following steps as shown in FIG. 6.

[0154] S601, the main AI server determines the CV(r, r+1, s, F)th communication volume.

[0155] In the embodiment of the present application, the CV(r, r+1, s, F)th communication volume is the communication volume of the rth training device transmitting the first feature vector corresponding to the s th batch to the r+1th training device.

[0156] S602, the main AI server determines the Speed(r, r+1)th bandwidth speed.

[0157] In the embodiment of the present application, the Speed(r, r+1)th bandwidth speed is the bandwidth speed of the rth training device transmitting data to the r+1th training device. The Speed(r, r+1)th bandwidth speed can be determined according to the model of the training device, and the corresponding bandwidth speed of different models of training devices is not the same.

[0158] S603, the main AI server determines the CT(r, r+1, s, F)th forward communication time length according to the CV(r, r+1, s, F)th communication volume and the Speed(r, r+1)th bandwidth speed.

[0159] In the embodiment of the present application, the main AI server takes the ratio of the CV(r, r+1, s, F)th communication volume to the Speed(r, r+1)th bandwidth speed as the CT(r, r+1, s, F)th forward communication time length.

[0160] Embodiment C2, the main AI server can directly measure the CT(r, r+1, s, F)th forward communication time length.

[0161] In one possible implementation, the measurement method can be that the main AI server actually calls the rth training device to transmit the first feature vector corresponding to the s th batch to the r+1th training device, calculates the time used, and the time used is the CT(r, r+1, s, F)th forward communication time length.

[0162] Through the above-mentioned embodiment C1 or embodiment C2, the main AI server can quickly determine the CT(r, r+1, s, F)th forward communication time length.

[0163] For example, the CT(r+1, r, s, B) represents the rth training device, the r+1 represents the r+1th training device, the s represents the st batch, and the B represents the back propagation. The r is an integer in [1, R], and the s is an integer in [1, S]. The CT(r+1, r, s, B) represents the back propagation time length of the st batch between the r+1th device and the rth device, or the back propagation time length of the st back propagation between the r+1th device and the rth device, or the communication time length of the r+1th training device to the rth training device for transmitting the second feature vector corresponding to the st batch. The second feature vector is the back propagation feature vector corresponding to the first network layer in the N(r+1) network layers allocated to the r+1th training device, and the N(r+1) network layers are in the N network layers, and N(r+1) is an integer less than or equal to N.

[0164] The main AI server can determine the CT(r+1, r, s, B) by two possible implementation manners, including an implementation manner D1 and an implementation manner D2.

[0165] The implementation manner D1 includes the following steps as shown in FIG. 7.

[0166] S701, the main AI server determines the CV(r+1, r, s, B).

[0167] In the embodiment of the present application, the CV(r+1, r, s, B) is the communication amount of the r+1th training device to the rth training device for transmitting the second feature vector corresponding to the st batch.

[0168] S702, the main AI server determines the Speed(r+1, r).

[0169] In the embodiment of the present application, the Speed(r+1, r) is the bandwidth speed of the r+1th training device to the rth training device for transmitting data. The Speed(r+1, r) can be determined according to the model of the training device, and the corresponding bandwidth speed of different models of training devices is not the same.

[0170] S703, the main AI server determines the CT(r+1, r, s, B) according to the CV(r+1, r, s, B) and the Speed(r+1, r).

[0171] In the embodiments of the present application, the main AI server takes the ratio of the CV(r+1, r, s, B)th communication volume to the Speed(r+1, r)th bandwidth speed as the CT(r+1, r, s, B)th reverse communication time length.

[0172] In an embodiment D2, the main AI server can directly measure the CT(r+1, r, s, B)th reverse communication time length.

[0173] In a possible implementation, the measurement method can be that the main AI server actually calls the r+1th training device to transmit the second feature vector corresponding to the s th batch to the r th training device to calculate the used time length, and the used time length is the CT(r+1, r, s, B)th reverse communication time length.

[0174] Through the above-mentioned embodiment D1 or embodiment D2, the main AI server can quickly determine the CT(r+1, r, s, B)th reverse communication time length.

[0175] In S503, the main AI server determines a first execution order of the S forward tasks and the S reverse tasks on each training device in the R training devices according to the forward calculation time length of each forward task executed by each computing device, the reverse calculation time length of each reverse task, the communication time length of the forward task or the reverse task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device.

[0176] In the embodiments of the present application, the first execution order is an execution order in which the training time length of the R training devices in training the neural network model by using the S batches satisfies a first condition, and the first execution order includes at least one group of W2 continuous forward tasks, and W2 is less than or equal to W1.

[0177] In a possible implementation, the first condition includes the following two embodiments:

[0178] In a first embodiment, the first condition is that the training time length of the R training devices in training the neural network model by using the S batches according to the first execution order is the minimum.

[0179] In a second embodiment, the first condition is that when the R training devices train the neural network model by using the S batches according to the first execution order, the training time length of the neural network model is a non-minimum value, for example, the training time length of the neural network model can be a sub-minimum value.

[0180] By the method, in the case of limited training devices, the first execution order for quickly training the neural network model can be determined, and when the R training devices train the neural network model by using the S batches according to the first execution order, the training time length of the neural network model satisfies the first condition, so that the determined first execution order can reduce the idle state of the training devices, and the training efficiency of the neural network model is improved.

[0181] In the embodiment of the application, the main AI server can determine the first execution order of the S forward tasks and the S backward tasks on each training device in the R training devices through the following steps as shown in FIG. 8.

[0182] S801, the main AI server determines a target function and a constraint condition of the target function according to the forward calculation time length of each forward task executed by each computing device, the backward calculation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device.

[0183] In the embodiment of the application, the target function is that the training time length of the neural network model satisfies the first condition. For example, the target function is to minimize the training time length of the neural network model.

[0184] The following describes how to determine the target function in S801.

[0185] In one possible implementation, taking the i-th execution order of the S forward tasks and the S backward tasks on each training device as an example, the i-th execution order is any one of the multiple execution orders, and the training time length of the neural network model corresponding to the i-th execution order can be determined through the following steps as shown in FIG. 9.

[0186] S901, the main AI server determines the t(th) start time of the (1, 1, F).

[0187] In the embodiment of the application, the first 1 in t(th) indicates the first training device, the second 1 indicates the first batch, and F indicates forward calculation. The t(th) start time of (1, 1, F) can be represented as the start time of the first training device performing forward calculation on the first batch according to the i-th execution order, or the t(th) start time of (1, 1, F) can also be represented as the start time of the first forward task on the first training device under the i-th execution order. The t(th) start time of (1, 1, F) can be regarded as the start time of model training.

[0188] It should be understood that the t(th) starting time point (1, 1, F) can be set as a time point corresponding to a training duration of the neural network model corresponding to the i(th) execution order determined by the master AI server using the steps shown in FIG. 9, or can be set as 0, rather than the starting time point of the actual forward calculation of the first batch by the first training device using the i(th) execution order.

[0189] S902, the master AI server determines the t(th) ending time point (1, S, B) according to the t(th) starting time point (1, 1, F), the forward calculation duration of each forward task performed by each training device, the backward calculation duration of each backward task performed by each training device, and the communication duration of the forward task or the backward task corresponding to the same batch between two training devices.

[0190] In the embodiments of the present application, 1 in t(th) ending time point (1, S, B) represents the first training device, S represents the S(th) batch, and B represents backward calculation. The t(th) ending time point (1, S, B) can represent the ending time point of the backward calculation of the S(th) batch by the first training device using the i(th) execution order, or the t(th) ending time point (1, S, B) can also represent the ending time point of the S(th) backward task on the first training device under the i(th) execution order. The t(th) ending time point (1, S, B) can be regarded as the ending time point of the model training.

[0191] It should be understood that the t(th) ending time point (1, S, B) is the estimated ending time point of the backward calculation of the S(th) batch by the first training device using the i(th) execution order by the master AI server, rather than the actual ending time point of the backward calculation of the S(th) batch by the first training device using the i(th) execution order.

[0192] Any starting time point or any ending time point determined subsequently is the time point estimated by the master AI server, rather than the actual calculation time point of the training device, which will not be described below.

[0193] S903, the master AI server determines the training duration of the neural network model corresponding to the i(th) execution order based on the t(th) starting time point (1, 1, F) and the t(th) ending time point (1, S, B).

[0194] In the embodiments of the present application, the master AI server takes the difference between the t(th) ending time point (1, S, B) and the t(th) starting time point (1, 1, F) as the training duration of the neural network model corresponding to the i(th) execution order.

[0195] It should be understood that if 0 is taken as the t(th) starting time point (1, 1, F), after the t(th) ending time point (1, S, B) is determined through the steps shown in FIG. 9, the difference between the t(th) ending time point (1, S, B) and 0 can be taken as the training duration of the neural network model corresponding to the i(th) execution order.

[0196] If the step shown in FIG. 9 is performed at the current time, the current time can be taken as the t(th) (1, 1, F)thstart time, and after the t(th) (1, S, B)thend time is determined through the step shown in FIG. 9, the difference between the t(th) (1, S, B)thend time and the current time can be taken as the training duration of the neural network model corresponding to the i(th) execution order.

[0197] FIG. 10 shows a structure diagram of training a neural network model in a pipeline parallel scheme of the i(th) execution order. In the i(th) execution order shown in FIG. 10, the order of performing forward calculation and backward calculation on the 5 batches by the 1(st) training device is: performing forward calculation on the 1(st) batch, performing forward calculation on the 2(nd) batch, performing forward calculation on the 3(rd) batch, performing forward calculation on the 4(th) batch, performing forward calculation on the 5(th) batch, performing backward calculation on the 1(st) batch, performing backward calculation on the 2(nd) batch, performing backward calculation on the 3(rd) batch, performing backward calculation on the 4(th) batch, and performing backward calculation on the 5(th) batch. The order of performing forward calculation and backward calculation on the 5 batches by other training devices refers to that of the 1(st) training device, which will not be repeated here.

[0198] The neural network model to be trained includes 100 network layers, and the training device cluster includes 4 training devices, which are the 1(st) training device, the 2(nd) training device, the 3(rd) training device, and the 4(th) training device. The 100 network layers are allocated to the 4 training devices, and it is set that the 1(st) training device is allocated N(1) network layers, the 2(nd) training device is allocated N(2) network layers, the 3(rd) training device is allocated N(3) network layers, and the 4(th) training device is allocated N(4) network layers. Wherein, the sum of N(1), N(2), N(3), and N(4) is 100.

[0199] The training set includes 1000 training data. If the 1000 training data is divided into 5 batches, it is set that the 1(st) batch includes M(1) training data, the 2(nd) batch includes M(2) training data, the 3(rd) batch includes M(3) training data, the 4(th) batch includes M(4) training data, and the 5(th) batch includes M(5) training data. Wherein, the sum of M(1), M(2), M(3), M(4), and M(5) is 1000.

[0200] The main AI server can determine the t(th) (1, 1, F)th start time and the t(th) (1, 5, B)th end time through the steps in FIG. 9, where the t(th) (1, 1, F)th start time is a start time of the first training device performing forward calculation on the first batch using the i(th) execution sequence, the t(th) (1, 5, B)th end time is an end time of the first training device performing backward calculation on the fifth batch using the i(th) execution sequence, and a difference between the t(th) (1, 5, B)th end time and the t(th) (1, 1, F)th start time is taken as a training duration of the neural network model corresponding to the i(th) execution sequence.

[0201] It should be understood that the t(th) (1, 1, F)th start time and the t(th) (1, 5, B)th end time are both estimated times of the main AI server, and are not actual calculation times of the training device.

[0202] The present application provides a possible implementation, which can determine the t(th) (1, S, B)th end time in S902 through the following steps as shown in FIG. 11.

[0203] S1101, the main AI server determines the t(th) (1, S, B)th start time.

[0204] In the embodiment of the present application, the t(th) (1, S, B)th start time is a start time of the first training device performing backward calculation on the S(th) batch using the i(th) execution sequence.

[0205] In a possible implementation, the t(th) (1, S, B)th start time can be determined based on the t(th) (r, s, B)th end time, where the t(th) (r, s, B)th end time is an end time of the r(th) training device performing backward calculation on the s(th) batch using the i(th) execution sequence, and the t(th) (r, s, B)th end time is determined according to the t(th) (1, 1, F)th start time, the T(th) (r, s, F)th forward calculation duration and the T(th) (r, s, B)th backward calculation duration, r is an integer taking values in [1, R], and s is an integer taking values in [1, S].

[0206] For the t(th) (r, s, B)th end time, the following describes how the t(th) (r, s, B)th end time is determined.

[0207] The t(r, s, B)th end time is determined according to the t(r, s, B)th start time and the T(r, s, B)th reverse calculation duration. Specifically, the main AI server can take the sum of the t(r, s, B)th start time and the T(r, s, B)th reverse calculation duration as the t(r, s, B)th end time. The t(r, s, B)th start time is the start time of the rth training device performing reverse calculation on the s th batch by using the i th execution sequence. The T(r, s, B)th reverse calculation duration is the duration of the rth training device performing reverse calculation on the s th batch by using the i th execution sequence. The determination of the T(r, s, B)th reverse calculation duration has been described in detail above, and will not be repeated here.

[0208] The following describes how to determine the t(r, s, B)th start time, including the following two cases, including case E1 and case E2.

[0209] Case E1, in the case where the rth training device is in a cooling phase at the t(r, s, B)th start time, the cooling phase is a phase in which the rth training device continuously performs reverse calculation. The t(r, s, B)th start time can be determined by the following steps shown in FIG. 12.

[0210] S1201, the main AI server determines the t(r, s-1, B)th end time and the t(r+1, s, B)th end time.

[0211] In the embodiments of the present application, the t(r, s-1, B)th end time is the end time of the rth training device performing reverse calculation on the s-1th batch by using the i th execution sequence, and the t(r+1, s, B)th end time is the end time of the r+1th training device performing reverse calculation on the s th batch by using the i th execution sequence.

[0212] S1202, the main AI server determines the t(r, s, B)th start time based on the t(r, s-1, B)th end time and the t(r+1, s, B)th end time.

[0213] For the method of determining the t(r, s, B)th start time in S1202, the present application provides the following two possible implementation manners, including implementation manner F1 and implementation manner F2.

[0214] Implementation manner F1, the main AI server can determine the t(r, s, B)th start time by the following steps shown in FIG. 13.

[0215] S1301, the main AI server determines the ref(r, s, B)th reference moment according to the t(r, s-1, B)th end moment and the t(r+1, s, B)th end moment.

[0216] In the embodiment of the application, the ref(r, s, B)th reference moment is the maximum end moment of the t(r, s-1, B)th end moment and the t(r-1, s, B)th end moment, that is, the ref(r, s, B)th reference moment = max(t(r, s-1, B)th end moment, t(r-1, s, B)th end moment), wherein max represents taking the maximum value.

[0217] S1302, the main AI server determines the CT(r+1, r, s, B)th reverse communication duration.

[0218] In the embodiment of the application, the CT(r+1, r, s, B)th reverse communication duration is the communication duration of the r+1th training device transmitting the second feature vector corresponding to the s th batch to the r th training device, wherein the second feature vector is determined according to the reverse calculation feature vector corresponding to the first network layer in the N(r+1) network layers allocated to the r+1th training device, and the N network layers include N(r+1) network layers, and N(r+1) is an integer less than or equal to N.

[0219] S1303, the main AI server determines the t(r, s, B)th start moment according to the ref(r, s, B)th reference moment and the CT(r+1, r, s, B)th reverse communication duration.

[0220] In the embodiment of the application, the main AI server takes the sum of the ref(r, s, B)th reference moment and the CT(r+1, r, s, B)th reverse communication duration as the t(r, s, B)th start moment.

[0221] The method shown in the above FIG. 13 provides a method for determining the t(r, s, B)th start moment.

[0222] Embodiment F2, the main AI server can determine the t(r, s, B)th start moment through the following steps as shown in FIG. 14.

[0223] S1401, the main AI server determines the ref(r, s, B)th reference moment according to the t(r, s-1, B)th end moment and the t(r+1, s, B)th end moment.

[0224] In the embodiments of the present application, the method for determining the ref(r, s, B)th reference time point by the main AI server in S1401 can refer to the method for determining the ref(r, s, B)th reference time point by the main AI server in S1301, which will not be repeated here.

[0225] S1402, in the case of the ref(r, s, B)th reference time point being the t(r, s-1, B)th end time point, the main AI server takes the ref(r, s, B)th reference time point as the t(r, s, B)th start time point.

[0226] S1403, in the case of the ref(r, s, B)th reference time point being the t(r+1, s, B)th end time point, the main AI server determines the CV(r+1, r, s, B)th reverse communication duration, and determines the t(r, s, B)th start time point according to the ref(r, s, B)th reference time point and the CT(r+1, r, s, B)th reverse communication duration.

[0227] In the embodiments of the present application, the method for determining the t(r, s, B)th start time point by the main AI server in S1403 can refer to the method for determining the t(r, s, B)th start time point by the main AI server in S1303, which will not be repeated here.

[0228] The method shown in the above Figure 14 provides a method for determining the t(r, s, B)th start time point.

[0229] In combination with the neural network model training process shown in Figure 10, it is explained in detail how to determine the training duration of the neural network model corresponding to the i th execution order. The main AI server needs to determine the t(1, 1, F)th start time point and the t(1, 5, B)th end time point, wherein the t(1, 5, B)th end time point is determined according to the t(1, 5, B)th start time point and the T(1, 5, B)th reverse calculation duration, and the three satisfy the following relationship:

[0230] t(1, 5, B)th end time point = t(1, 5, B)th start time point + T(1, 5, B)th reverse calculation duration

[0231] Wherein, the t(1, 5, B)th end time point is the end time point of the first training device adopting the i th execution order to execute the reverse calculation of the 5th batch, the t(1, 5, B)th start time point is the start time point of the first training device adopting the i th execution order to execute the reverse calculation of the 5th batch, and the T(1, 5, B)th reverse calculation duration is the duration of the first training device executing the reverse calculation of the 5th batch.

[0232] According to the time of performing the reverse calculation continuously by the four training devices in FIG. 10, the cooling stage corresponding to the four training devices can be determined as the area marked in FIG. 15.

[0233] Since the first training device is in the cooling stage at the t(1,5,B)th start time, the main AI server can first determine the t(1,4,B)th end time and the t(2,5,B)th end time, and determine the t(1,5,B)th start time according to the t(1,4,B)th end time and the t(2,5,B)th end time by referring to the steps in FIG. 13 or FIG. 14.

[0234] Wherein, the main AI server can determine the t(1,4,B)th end time by referring to the way of determining the t(1,5,B)th end time, the main AI server can also determine the t(2,5,B)th end time by referring to the way of determining the t(1,5,B)th end time, and so on, which will not be repeated here.

[0235] In case E2, when the rth training device is in the stable stage at the t(r,s,B)th start time, the stable stage is the stage in which the rth training device alternately performs forward calculation and reverse calculation, or the stable stage can also be the stage in which the rth training device alternately performs at least one forward calculation and at least one reverse calculation, or the stable stage is other stage except the start stage and the cooling stage, the t(r,s,B)th start time can be determined by the following steps shown in FIG. 16.

[0236] S1601, the main AI server determines the t(r,x1,F)th end time / t(r,x2,B)th end time and the t(r+1,s,B)th end time.

[0237] In the embodiment of the present application, the t(r+1,s,B)th end time is the end time of performing the reverse calculation on the s th batch by the r+1th training device using the i th execution sequence.

[0238] The t(r,x2,B)th end time is the end time of performing the reverse calculation on the x2th batch by the rth training device using the i th execution sequence, the reverse calculation of the x2th batch in the rth training device is adjacent to and before the reverse calculation of the s th batch, and x2 is less than s.

[0239] It should be understood that the determination of the t(r+1,s,B)th end time and the t(r,x2,B)th end time can refer to the t(r,s,B)th end time, which will not be repeated here.

[0240] The t(r, x1, F)th end time point is an end time point of forward calculation of the x1th batch by the rth training device using the ith execution sequence, and the forward calculation of the x1th batch in the rth training device is adjacent to the backward calculation of the sth batch.

[0241] The t(r, x1, F)th end time point is determined as follows.

[0242] The t(r, x1, F)th end time point is determined according to the t(r, x1, F)th start time point and the T(r, x1, F)th forward calculation duration. Specifically, the main AI server can take the sum of the t(r, x1, F)th start time point and the T(r, x1, F)th forward calculation duration as the t(r, x1, F)th end time point. The t(r, x1, F)th start time point is a start time point of forward calculation of the x1th batch by the rth training device using the ith execution sequence, and the T(r, x1, F)th forward calculation duration is a duration of forward calculation of the x1th batch by the rth training device. The determination of the T(r, x1, F)th forward calculation duration can refer to the determination of the T(r, s, F)th forward calculation duration, which is not repeated here.

[0243] The t(r, x1, F)th start time point is determined as follows. The present application provides two cases, including case G1 and case G2.

[0244] In case G1, when the rth training device is in the stable stage at the t(r, x1, F)th start time point, the t(r, x1, F)th start time point can be determined by the following steps shown in FIG. 17.

[0245] S1701, the main AI server determines the t(r-1, x1, F)th end time point and the t(r, y1, B)th end time point / the t(r, y2, F)th end time point.

[0246] In the embodiment of the present application, the t(r-1, x1, F)th end time point is an end time point of forward calculation of the x1th batch by the r-1th training device using the ith execution sequence, and the determination of the t(r-1, x1, F)th end time point can refer to the t(r, x1, F)th end time point, which is not repeated here.

[0247] The t(r, y1, B)th end time point is an end time point of performing back propagation on the y1th batch by the rth training device using the ith execution sequence, the back propagation of the y1th batch in the rth training device is adjacent to the forward propagation of the x1th batch and before the forward propagation of the x1th batch, and y1 is less than x1. The t(r, y1, B)th end time point can be determined with reference to the t(r, x1, F)th end time point, and details are not described herein.

[0248] The t(r, y2, F)th end time point is an end time point of performing forward propagation on the y2th batch by the rth training device using the ith execution sequence, the forward propagation of the y2th batch in the rth training device is adjacent to the forward propagation of the x1th batch and before the forward propagation of the x1th batch, and y2 is less than x1. The t(r, y2, F)th end time point can be determined with reference to the t(r, x1, F)th end time point, and details are not described herein.

[0249] In S1702, the main AI server determines the t(r, x1, F)th start time point based on the t(r-1, x1, F)th end time point and the t(r, y1, B)th end time point / t(r, y2, F)th end time point.

[0250] In the embodiments of the present application, if the back propagation of the y1th batch in the rth training device is adjacent to the forward propagation of the x1th batch and before the forward propagation of the x1th batch, the main AI server determines the t(r, x1, F)th start time point according to the t(r-1, x1, F)th end time point and the t(r, y1, B)th end time point. The main AI server can determine the t(r, x1, F)th start time point with reference to the step in S1202 in FIG. 12, and details are not described herein.

[0251] If the forward propagation of the y2th batch in the rth training device is adjacent to the forward propagation of the x1th batch and before the forward propagation of the x1th batch, the main AI server determines the t(r, x1, F)th start time point according to the t(r-1, x1, F)th end time point and the t(r, y2, F)th end time point. The main AI server can determine the t(r, x1, F)th start time point with reference to the step in S1202 in FIG. 12, and details are not described herein.

[0252] A method for determining the t(r, x1, F)th start time point is provided by the method shown in FIG. 17.

[0253] Taking the neural network model training process shown in FIG. 10 as an example, after the start stage and the cooling stage, four stable stages corresponding to four training devices can be determined as the regions marked in FIG. 18.

[0254] Taking the t(th) start time point of the 3rd training device as an example, since the 3rd training device is in the stable stage at the t(th) start time point of the 3rd training device, the main AI server first determines the t(th) end time point of the 5th training device and the t(th) end time point of the 2nd training device.

[0255] The t(th) end time point of the 4th training device is determined according to the t(th) start time point of the 4th training device and the T(th) reverse calculation duration, since the 4th training device is in the stable stage at the t(th) start time point of the 4th training device, the main AI server can determine the t(th) start time point of the 4th training device by referring to the steps shown in FIG. 16.

[0256] The t(th) end time point of the 3rd training device is determined according to the t(th) start time point of the 3rd training device and the T(th) forward calculation duration, since the 3rd training device is in the stable stage at the t(th) start time point of the 3rd training device, the main AI server can determine the t(th) start time point of the 3rd training device by referring to the steps shown in FIG. 17.

[0257] In case G2, when the rth training device is in the start-up stage at the t(th) start time point of the rth training device, the start-up stage is a stage in which the rth training device continuously performs forward calculation, the t(th) start time point of the rth training device can be determined by the following steps shown in FIG. 19.

[0258] S1901, the main AI server determines the t(th) end time point of the (r-1)th training device and the t(th) end time point of the (r, x1-1)th training device.

[0259] In the embodiments of the present application, the t(th) end time point of the (r-1)th training device is the end time point of the r-1th training device performing forward calculation on the x1th batch by using the i-th execution sequence, and the t(th) end time point of the rth training device is the end time point of the rth training device performing forward calculation on the x1-1th batch by using the i-th execution sequence.

[0260] The determination of the t(th) end time point of the (r-1)th training device can refer to the t(th) end time point of the rth training device, which will not be repeated here.

[0261] The determination of the t(th) end time point of the rth training device can refer to the t(th) end time point of the rth training device, which will not be repeated here.

[0262] S1902, the main AI server determines the t(th) start time point of the rth training device based on the t(th) end time point of the (r-1)th training device and the t(th) end time point of the (r, x1-1)th training device.

[0263] In the embodiment of the present application, the main AI server determines the t(r, x1, F)th start time. For details, refer to S1202 in FIG. 12, which will not be repeated here.

[0264] A method for determining the t(r, x1, F)th start time is provided by the method shown in FIG. 19.

[0265] For example, in combination with the neural network model training process shown in FIG. 10, according to the stages in which each of the four training devices successively performs forward calculation, it can be determined that the start stages corresponding to the four training devices are the areas labeled in FIG. 20.

[0266] For example, at the t(2, 3, F)th start time, since the second training device is in the start stage at the t(2, 3, F)th start time, the main AI server first determines the t(2, 2, F)th end time and the t(1, 3, F)th end time.

[0267] The t(2, 2, F)th end time is determined according to the t(2, 2, F)th start time and the T(2, 2, F)th forward calculation duration. Since the second training device is in the start stage at the t(2, 2, F)th start time, the main AI server can refer to the steps shown in FIG. 19 to determine the t(2, 2, F)th start time.

[0268] The t(1, 3, F)th end time is determined according to the t(1, 3, F)th start time and the T(1, 3, F)th forward calculation duration. Since the first training device is in the start stage at the t(1, 3, F)th start time, the main AI server can refer to the steps shown in FIG. 19 to determine the t(1, 3, F)th start time.

[0269] S1602, the main AI server determines the t(r, s, B)th start time based on the t(r, x1, F)th end time / t(r, x2, B)th end time and the t(r+1, s, B)th end time.

[0270] In the embodiment of the present application, if the forward calculation of the x1th batch in the rth training device is located before and adjacent to the backward calculation of the sth batch, the main AI server determines the t(r, s, B)th start time according to the t(r, x1, F)th end time and the t(r+1, s, B)th end time. The main AI server can specifically refer to the steps in S1202 in FIG. 12 to determine the t(r, s, B)th start time.

[0271] If the backward calculation of the x2th batch in the rth training device is located before and adjacent to the backward calculation of the s th batch, the master AI server determines the t(r,s,B)th start time according to the t(r,x2,B)th end time and the t(r+1,s,B)th end time. The master AI server can specifically refer to the steps in S1202 in FIG. 12 to determine the t(r,s,B)th start time.

[0272] The method shown in FIG. 16 provides a method for determining the t(r,s,B)th start time.

[0273] In the embodiment of the present application, the T(1,S,B)th backward calculation duration is the duration of the first training device performing backward calculation on the S th batch by using the i th execution sequence. The determination manner of the T(1,S,B)th backward calculation duration can refer to the determination manner of the T(r,s,B)th backward calculation duration, which will not be repeated here.

[0274] In the embodiment of the present application, the T(1,S,B)th backward calculation duration is the duration of the first training device performing backward calculation on the S th batch by using the i th execution sequence. The determination manner of the T(1,S,B)th backward calculation duration can refer to the determination manner of the T(r,s,B)th backward calculation duration, which will not be repeated here.

[0275] S1103, the master AI server determines the t(1,S,B)th end time based on the t(1,S,B)th start time and the T(1,S,B)th backward calculation duration.

[0276] In the embodiment of the present application, the master AI server takes the sum of the t(1,S,B)th start time and the T(1,S,B)th backward calculation duration as the t(1,S,B)th end time.

[0277] The method shown in FIG. 11 provides a method for quickly determining the t(1,S,B)th end time.

[0278] The following will introduce in detail how to determine the constraint condition of the objective function in S801.

[0279] In a possible implementation manner, taking the i th execution sequence of the S forward tasks and the S backward tasks on each training device as an example, the i th execution sequence is any one of the multiple execution sequences, and the constraint condition met by the i th execution sequence includes at least one of the following:

[0280] 1) The end time of the s th forward task on the r th training device is less than the start time of the s th forward task on the r+1 th training device, r is an integer taking all values in [1,R], and s is an integer taking all values in [1,S].

[0281] In the embodiments of the present application, the above constraint condition can also be represented as: the end time of the rth training device performing forward calculation on the s-th batch is less than the start time of the r+1th training device performing forward calculation on the s-th batch.

[0282] The above constraint condition can also be represented as: the sum of the end time of the rth training device performing forward calculation on the s-th batch and the CT(r, r+1, s, F)th forward communication duration is less than or equal to the start time of the r+1th training device performing forward calculation on the s-th batch.

[0283] For example, in combination with the neural network model training process shown in FIG. 10, the end time t(2, 3, F) is less than the start time t(3, 3, F), where the end time t(2, 3, F) is the end time of the 2nd training device performing forward calculation on the 3rd batch using the i-th execution order, and the start time t(3, 3, F) is the start time of the 3rd training device performing forward calculation on the 3rd batch using the i-th execution order.

[0284] 2) The end time of the s-th forward task on the rth training device is less than or equal to the start time of the s+1th forward task on the rth training device.

[0285] In the embodiments of the present application, the above constraint condition can also be represented as: the end time of the rth training device performing forward calculation on the s-th batch is less than or equal to the start time of the r+1th training device performing forward calculation on the s-th batch.

[0286] For example, in combination with the neural network model training process shown in FIG. 10, the end time t(2, 3, F) is less than or equal to the start time t(2, 4, F), where the end time t(2, 3, F) is the end time of the 2nd training device performing forward calculation on the 3rd batch using the i-th execution order, and the start time t(2, 4, F) is the start time of the 2nd training device performing forward calculation on the 4th batch using the i-th execution order.

[0287] 3) The end time of the s-th backward task on the r+1th training device is less than the start time of the s-th backward task on the rth training device.

[0288] In the embodiments of the present application, the above constraint condition can also be represented as: the end time of the r+1th training device performing backward calculation on the s-th batch is less than the start time of the rth training device performing backward calculation on the s-th batch.

[0289] The constraint condition can also be expressed as: the sum of the end moment of the r+1th training device performing the backward calculation on the s th batch and the CT(r+1, r, s, B)th backward communication duration is less than or equal to the start moment of the rth training device performing the backward calculation on the s th batch.

[0290] For example, in combination with the neural network model training process shown in FIG. 10, the t(2, 3, B)th end moment is less than the t(1, 3, B)th start moment, where the t(2, 3, B)th end moment is the end moment of the 2nd training device performing the forward calculation on the 3rd batch in the i th execution order, and the t(1, 3, B)th start moment is the start moment of the 1st training device performing the backward calculation on the 3rd batch in the i th execution order.

[0291] 4) The end moment of the s th backward task on the r th training device is less than or equal to the start moment of the s+1th backward task on the r th training device.

[0292] In the embodiment of the present application, the constraint condition can also be expressed as: the end moment of the r th training device performing the backward calculation on the s th batch is less than or equal to the start moment of the r th training device performing the backward calculation on the s+1th batch.

[0293] For example, in combination with the neural network model training process shown in FIG. 10, the t(2, 3, B)th end moment is less than or equal to the t(2, 4, B)th start moment, where the t(2, 3, B)th end moment is the end moment of the 2nd training device performing the backward calculation on the 3rd batch in the i th execution order, and the t(2, 4, B)th start moment is the start moment of the 2nd training device performing the backward calculation on the 4th batch in the i th execution order.

[0294] 5) The end moment of the s th forward task on the r th training device is less than or equal to the start moment of the s th backward task on the r th training device.

[0295] In the embodiment of the present application, the constraint condition can also be expressed as: the end moment of the r th training device performing the forward calculation on the s th batch is less than or equal to the start moment of the r th training device performing the backward calculation on the s th batch.

[0296] For example, in combination with the neural network model training process shown in FIG. 10, the end time of the t(th) (2, 3, F) is less than or equal to the start time of the t(th) (2, 3, B), where the end time of the t(th) (2, 3, F) is the end time of the forward calculation of the third batch by the second training device using the i(th) execution sequence, and the start time of the t(th) (2, 3, B) is the start time of the backward calculation of the third batch by the second training device using the i(th) execution sequence.

[0297] 6) In the case where the s1(th) forward task is executed before the s2(th) backward task, the end time of the s1(th) forward task on each training device is less than or equal to the start time of the s2(th) backward task; in the case where the s2(th) backward task is executed before the s1(th) forward task, the end time of the s2(th) backward task on each training device is less than or equal to the start time of the s1(th) forward task, s1 is an integer taking values in [1, S], s2 is an integer taking values in [1, S], and s1 is not equal to s2.

[0298] In the embodiments of the present application, the above constraint condition can also be expressed as: in the case where the forward calculation corresponding to the s1(th) batch is executed before the backward calculation corresponding to the s2(th) batch, the end time of the forward calculation of the s1(th) batch on each training device is less than or equal to the start time of the backward calculation of the s2(th) batch; in the case where the backward calculation corresponding to the s2(th) batch is executed before the forward calculation corresponding to the s1(th) batch, the end time of the backward calculation of the s2(th) batch on each training device is less than or equal to the start time of the forward calculation of the s1(th) batch, s1 is an integer taking values in [1, S], s2 is an integer taking values in [1, S], and s1 is not equal to s2.

[0299] For example, in combination with the neural network model training process shown in FIG. 10, the end time of the t(th) (3, 2, F) is less than or equal to the start time of the t(th) (3, 3, B) in the third training device.

[0300] In the third training device, the backward calculation corresponding to the first batch is executed before the forward calculation corresponding to the fourth batch, so the end time of the t(th) (3, 1, B) is less than or equal to the start time of the t(th) (3, 4, F).

[0301] 7) The number of batches executed continuously in the r(th) training device is less than or equal to W1.

[0302] S802, the main AI server uses a linear solver to solve the objective function according to the constraint condition, and obtains the first execution sequence of the S forward tasks and the S backward tasks on each training device.

[0303] In the embodiments of the present application, the linear solver can be generated according to any one of the branch and bound method, the cutting plane method, etc., which is not limited herein.

[0304] It should be understood that the main AI server can select the first execution order from a plurality of execution orders by using the linear solver, and the first execution order determined by the main AI server satisfies the constraint condition in S801.

[0305] In the steps shown in FIG. 8, the main AI server can quickly obtain the first execution order by using the linear solver to solve the objective function according to the constraint condition.

[0306] Through the steps shown in FIG. 5, the main AI server can determine a first execution order for model training optimization, wherein the first execution order is an execution order in which the training time length of the R training devices for training the neural network model using the S batches satisfies the first condition, and the first execution order includes at least one group of W2 continuous executed forward tasks.

[0307] After that, the R training devices can train the neural network model using the S batches according to the first execution order, and when the R training devices train the neural network model using the S batches according to the first execution order, the training time length of the neural network model satisfies the first condition.

[0308] During the process of the R training devices training the neural network model using the S batches according to the first execution order, the main AI server can also establish a forward communication domain for the forward tasks on the R training devices, and the main AI server can also establish a backward communication domain for the backward tasks on the R training devices.

[0309] The following explains in detail how to establish a forward communication domain for the rth training device in the R training devices, which can perform the following steps shown in FIG. 21, wherein r is an integer that traverses [1, R]:

[0310] S2101, the main AI server executes the st forward task through the rth training device to obtain a first feature vector corresponding to the st forward task.

[0311] In the embodiments of the present application, the first feature vector is a forward calculation feature vector corresponding to the last network layer in the N(r) network layers allocated to the rth training device.

[0312] S2102, the main AI server establishes a forward communication domain between the rth training device and the r+1th training device.

[0313] In the embodiments of the present application, the forward communication domain between the rth training device and the r+1th training device is used to send the first feature vector corresponding to the st forward task to the r+1th training device.

[0314] In a possible implementation, after the master AI server establishes the forward communication domain between the rth training device and the (r+1)th training device, the rth training device can send the first feature vector corresponding to the s-th forward task to the (r+1)th training device through the forward communication domain between the rth training device and the (r+1)th training device.

[0315] In a possible implementation, the forward communication domain between the rth training device and the (r+1)th training device can be established in the following manner: the rth training device can send a first request message to the (r+1)th training device, where the first request message is used to request establishment of the forward communication domain between the rth training device and the (r+1)th training device. After receiving the first request message, the (r+1)th training device can send a first response message to the rth training device, where the first response message is used to indicate that the forward communication is allowed, or the first response message is used to indicate that the forward communication domain has been established. After receiving the first response message, the rth training device determines that the forward communication domain between the rth training device and the (r+1)th training device has been established.

[0316] In the method shown in FIG. 21, when the rth training device needs to transmit the first feature vector corresponding to the s-th forward task to the (r+1)th training device during execution of the s-th forward task by the rth training device, an independent forward communication domain can be immediately established between the rth training device and the (r+1)th training device, and the first feature vector corresponding to the s-th forward task is transmitted using the forward communication domain, which is beneficial to improving the transmission efficiency of the first feature vector corresponding to the s-th forward task, and further beneficial to improving the training efficiency of the neural network model.

[0317] The following explains in detail how to establish a backward communication domain for the rth training device in R training devices, and the following steps shown in FIG. 22 can be executed, where r is an integer that traverses [1, R]:

[0318] S2201, the master AI server executes the s-th backward task through the rth training device to obtain a second feature vector corresponding to the s-th backward task.

[0319] In the embodiment of the present application, the second feature vector is a backward calculation feature vector corresponding to the first network layer in the N(r) network layers allocated to the rth training device.

[0320] S2202, the master AI server establishes a backward communication domain between the rth training device and the (r-1)th training device.

[0321] In the embodiment of the present application, the reverse communication domain between the rth training device and the (r-1)th training device is used to send the second feature vector corresponding to the s-th reverse task to the (r-1)th training device.

[0322] In a possible implementation, after the main AI server establishes the reverse communication domain between the rth training device and the (r-1)th training device, the rth training device can send the second feature vector corresponding to the s-th reverse task to the (r-1)th training device through the reverse communication domain between the rth training device and the (r-1)th training device.

[0323] In a possible implementation, the reverse communication domain between the rth training device and the (r-1)th training device can be established in the following manner: the rth training device can send a second request message to the (r-1)th training device, and the second request message is used to request the establishment of the reverse communication domain between the rth training device and the (r-1)th training device. After receiving the second request message, the (r-1)th training device can send a second response message to the rth training device, and the second response message is used to indicate that the reverse communication is allowed, or the second response message is used to indicate that the reverse communication domain has been established. After receiving the second response message, the rth training device determines that the reverse communication domain between the rth training device and the (r-1)th training device has been established.

[0324] In the method shown in FIG. 22, when the rth training device needs to transmit the second feature vector corresponding to the s-th reverse task to the (r-1)th training device during the execution of the s-th reverse task by the rth training device, an independent reverse communication domain can be immediately established between the rth training device and the (r-1)th training device, and the second feature vector corresponding to the s-th reverse task is transmitted using the reverse communication domain, which is beneficial to improve the transmission efficiency of the second feature vector corresponding to the s-th reverse task, and further beneficial to improve the training efficiency of the neural network model.

[0325] Currently, in the prior art, only one forward and reverse communication domain is established between two training devices, and the forward and reverse communication domain is established only after the execution of forward calculation and reverse calculation by the two training devices is completed, which will result in the inability to timely and quickly establish the forward and reverse communication domain, and the forward and reverse communication domain is used to transmit forward calculation feature vectors during forward calculation and to transmit reverse calculation feature vectors during reverse calculation, which will result in low transmission efficiency and further result in low training efficiency of the neural network model.

[0326] Through the method shown in FIGS. 21 and 22, it can be seen that two communication domains, i.e., a forward communication domain and a backward communication domain, can be established between the two training devices, the forward communication domain can be established after the training device ends forward calculation, and the backward communication domain can be established after the training device ends backward calculation, without waiting for both the forward calculation and the backward calculation of the two training devices to end before the communication domain can be established, so that the communication domain can be established in time and quickly. Since the forward communication domain is only used to transmit the forward calculation feature vector in the forward calculation process, and the backward communication domain is only used to transmit the backward calculation feature vector in the backward calculation process, it is beneficial to improve the transmission efficiency of the forward calculation feature vector and the backward calculation feature vector, and further beneficial to improve the training efficiency of the neural network model.

[0327] The following illustrates how to establish a forward-backward communication domain between two devices in the prior art by combining the 1F1B mode pipeline parallel scheme shown in FIG. 2, taking the example of the first training device performing forward calculation on the fifth batch and performing backward calculation on the second batch, and the second training device performing backward calculation on the second batch and performing forward calculation on the fifth batch, the corresponding amplified structure is shown in FIG. 23. The end time of the first training device performing forward calculation on the fifth batch is t1, at t1, the second training device is still performing backward calculation on the second batch, so the first training device cannot transmit the first feature vector corresponding to the fifth batch to the second training device at t1, and needs to wait until the second training device ends backward calculation on the second batch before the first training device can transmit the first feature vector corresponding to the fifth batch to the second training device.

[0328] The end time of the second training device performing backward calculation on the second batch is t2, at t2, the first training device ends forward calculation on the fifth batch, and the second training device ends backward calculation on the second batch, at this time, a forward-backward communication domain is established between the first training device and the second training device, and the first feature vector corresponding to the fifth batch and the second feature vector corresponding to the second batch are transmitted through the forward-backward communication domain at the same time. At t3, the transmission of the first feature vector corresponding to the fifth batch and the second feature vector corresponding to the second batch is completed, so the first training device performs backward calculation on the second batch at t3, and performs forward calculation on the fifth batch at t3.

[0329] The following illustrates how to establish two communication domains between two devices, perform forward calculation on the 4th batch by the 3rd training device and perform reverse calculation on the 2nd batch by the 3rd training device, perform reverse calculation on the 2nd batch by the 4th training device and perform forward calculation on the 4th batch by the 4th training device in the first execution order pipeline parallelism scheme shown in FIG. 10. The corresponding amplified structure is shown in FIG. 24. The end time of performing forward calculation on the 4th batch by the 3rd training device is t1. The 3rd training device can establish a forward communication domain between the 3rd training device and the 4th training device at t1, and transmit the first feature vector corresponding to the 4th batch to the 4th training device through the forward communication domain. At t2, the transmission of the first feature vector corresponding to the 4th batch is completed, so the 4th training device can perform forward calculation on the 4th batch at t2. Since the 4th training device is still performing reverse calculation on the 2nd batch at this time, the 4th training device can immediately perform forward calculation on the 4th batch after the reverse calculation on the 2nd batch is completed.

[0330] The end time of performing reverse calculation on the 2nd batch by the 4th training device is t3. The 4th training device can establish a reverse communication domain between the 4th training device and the 3rd training device at t3, and transmit the second feature vector corresponding to the 2nd batch to the 3rd training device through the reverse communication domain. At t4, the transmission of the second feature vector corresponding to the 2nd batch is completed, so the 3rd training device can perform reverse calculation on the 2nd batch at t4.

[0331] As can be seen from FIG. 24, between t3 and t4, the 4th training device transmits the second feature vector corresponding to the 2nd batch to the 3rd training device, and the 4th training device performs forward calculation on the 4th batch.

[0332] Based on the above embodiments, the embodiments of the present application also provide a model training optimization device. The device is applied to the main AI server shown in FIG. 3, and is used to implement the model training optimization method provided by the embodiments shown in FIG. 5. Referring to FIG. 25, the device includes an acquisition module 2501, a time length determination module 2502, an execution order determination module 2503, and a communication domain establishment module 2504. Wherein:

[0333] The acquisition module 2501 is configured to determine R training devices of a neural network model performing pipeline parallel training, S batches, and the number W1 of batches that can be continuously executed in each training device, wherein R is less than S, and R, S and W1 are all greater than 0.

[0334] The time length determination module 2502 is configured to determine the S forward tasks and the S backward tasks on each training device, the forward calculation time length of each of the S forward tasks, the backward calculation time length of each of the S backward tasks, and the communication time length of the forward task or the backward task corresponding to the same batch between two training devices when the S batches are trained on each training device.

[0335] The execution sequence determination module 2503 is configured to determine, according to the forward calculation time length of each forward task executed by each computing device, the backward calculation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, a first execution sequence of the S forward tasks and the S backward tasks on each training device in the R training devices, where the first execution sequence is an execution sequence in which the training time length of the R training devices in training the neural network model by using the S batches satisfies a first condition, and the first execution sequence includes at least one group of W2 continuous forward tasks, W2 is greater than 0 and less than or equal to W1.

[0336] In a possible implementation, the execution sequence determination module 2503 is specifically configured to: determine, according to the forward calculation time length of each forward task executed by each computing device, the backward calculation time length of each backward task, the communication time length of the forward task or the backward task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, an objective function and a constraint condition of the objective function; the objective function is that the training time length of the neural network model satisfies the first condition; and the objective function is solved by using a linear solver according to the constraint condition, to obtain the first execution sequence of the S forward tasks and the S backward tasks on each training device.

[0337] In a possible implementation, the constraint condition includes at least one of the following:

[0338] the end moment of the s-th forward task on the r-th training device is less than the start moment of the s-th forward task on the r+1-th training device, r is an integer that traverses [1, R], and s is an integer that traverses [1, S]; or

[0339] the end moment of the s-th forward task on the r-th training device is less than or equal to the start moment of the s+1-th forward task on the r-th training device; or

[0340] the end moment of the s-th backward task on the r+1-th training device is less than the start moment of the s-th backward task on the r-th training device; or

[0341] the end moment of the s-th backward task on the r-th training device is less than or equal to the start moment of the s+1-th backward task on the r-th training device; or

[0342] the end time of the s-th forward task on the r-th training device is less than or equal to the start time of the s-th backward task on the r-th training device; or

[0343] in a case where the s1-th forward task is executed before the s2-th backward task, the end time of the s1-th forward task on the r-th training device is less than or equal to the start time of the s2-th backward task; in a case where the s2-th backward task is executed before the s1-th forward task, the end time of the s2-th backward task on the r-th training device is less than or equal to the start time of the s1-th forward task, s1 is an integer taking values from 1 to S, s2 is an integer taking values from 1 to S, and s1 is not equal to s2; or

[0344] the number of batches continuously executed in the r-th training device is less than or equal to W1.

[0345] In a possible implementation, the first condition is that the training time length of the R training devices in training the neural network model by using the S batches in the first execution order is the minimum.

[0346] In a possible implementation, the communication domain establishing module 2504 is configured to:

[0347] establish a forward communication domain for forward tasks on the R training devices, and establish a backward communication domain for backward tasks on the R training devices.

[0348] In a possible implementation, the communication domain establishing module 2504 is specifically configured to:

[0349] For the r-th training device in the R training devices, the following steps are performed: performing, by the r-th training device, the s-th forward task to obtain a first feature vector corresponding to the s-th forward task, the first feature vector being a forward calculation feature vector corresponding to a last network layer allocated to the r-th training device; establishing a forward communication domain between the r-th training device and the r+1-th training device, the forward communication domain between the r-th training device and the r+1-th training device being used to send the first feature vector corresponding to the s-th forward task to the r+1-th training device.

[0350] In a possible implementation, the communication domain establishing module 2504 is specifically configured to:

[0351] The following steps are performed by the rth training device among the R training devices: obtaining a second feature vector corresponding to the st back task by performing the st back task by the rth training device, the second feature vector being a back calculation feature vector corresponding to a first network layer assigned to the rth training device; establishing a back communication domain between the rth training device and the r-1th training device, the back communication domain between the rth training device and the r-1th training device being used to send the second feature vector corresponding to the st back task to the r-1th training device.

[0352] In the embodiments of the present application, the acquisition module, the duration determination module, the execution sequence determination module and the communication domain establishment module can be implemented by software or by hardware. For example, the implementation of the execution sequence determination module is described below. Similarly, the implementation of the acquisition module, the duration determination module and the communication domain establishment module can refer to the implementation of the execution sequence determination module.

[0353] As an example of a software functional unit, the execution sequence determination module can include code running on a computing instance. The computing instance can include at least one of a physical host, a virtual machine, and a container. Further, the computing instance can be one or more. For example, the execution sequence determination module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0354] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to be set through a communication gateway in each VPC to realize interconnection between VPCs.

[0355] As an example of a module as a hardware functional unit, the execution order determination module can include at least one computing device such as a server or the like. Alternatively, the execution order determination module can also be a device implemented with an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the PLD can be implemented with a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0356] The multiple computing devices included in the execution order determination module can be distributed in the same region or in different regions. The multiple computing devices included in the execution order determination module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the execution order determination module can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0357] It should be noted that in other embodiments, the obtaining module can be used to determine R training devices of the neural network model performing pipeline parallel training, S batches, and the number W1 of batches that can be continuously executed in each training device. The duration determining module is used to determine S forward tasks and S reverse tasks when training S batches on each training device, the forward calculation duration of each forward task in S forward tasks, the reverse calculation duration of each reverse task in S reverse tasks, and the communication duration of the forward task or the reverse task corresponding to the same batch between two training devices. The execution order determining module can be used to determine the first execution order of S forward tasks and S reverse tasks on each training device in R training devices according to the forward calculation duration of each forward task executed by each computing device, the reverse calculation duration of each reverse task, the communication duration of the forward task or the reverse task corresponding to the same batch between two training devices, and the number W1 of batches that can be continuously executed in each training device, wherein the first execution order is an execution order in which the training duration of R training devices using S batches to train the neural network model satisfies a first condition, and the first execution order includes at least one group of W2 continuously executed forward tasks. The steps implemented by the obtaining module, the duration determining module, and the execution order determining module can be specified as needed, and the entire function of the model training optimization device can be realized by realizing different steps in the model training optimization method through the obtaining module, the duration determining module, and the execution order determining module, respectively.

[0358] The embodiments of the present application also provide a model training system, which includes one or more computing devices and one or more training devices.

[0359] The computing device is configured to: determine R training devices, S batches, and a number W1 of batches that can be continuously executed in each training device of a neural network model that is trained in a pipeline parallel manner, wherein R is less than S, and R, S, and W1 are all greater than 0; determine S forward tasks and S backward tasks when the S batches are trained on each training device, a forward computation duration of each forward task in the S forward tasks, a backward computation duration of each backward task in the S backward tasks, and a communication duration of a forward task or a backward task corresponding to the same batch of data between two training devices; determine, according to the forward computation duration of each forward task executed by each computing device, the backward computation duration of each backward task, the communication duration of the forward task or the backward task corresponding to the same batch of data between two training devices, and the number W1 of batches that can be continuously executed in each training device, a first execution order of the S forward tasks and the S backward tasks on each training device in the R training devices, wherein the first execution order is an execution order in which a training duration of the R training devices in training the neural network model using the S batches satisfies a first condition, and the first execution order includes at least one group of W2 continuous forward tasks, W2 is greater than 0 and less than or equal to W1.

[0360] The training device is configured to train the neural network model using the S batches according to the first execution order.

[0361] In a possible implementation, the computing device is specifically configured to: determine, according to the forward computation duration of each forward task executed by each computing device, the backward computation duration of each backward task, the communication duration of the forward task or the backward task corresponding to the same batch of data between two training devices, and the number W1 of batches that can be continuously executed in each training device, an objective function and a constraint condition of the objective function; the objective function is that the training duration of the neural network model satisfies the first condition; and a linear solver is used to solve the objective function according to the constraint condition to obtain the first execution order of the S forward tasks and the S backward tasks on each training device.

[0362] In a possible implementation, the computing device is specifically configured to: establish a forward communication domain for the forward tasks on the R training devices; and establish a backward communication domain for the backward tasks on the R training devices.

[0363] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute the model training optimization method.

[0364] The computer readable storage medium can be any available media or data storage device that can be accessed by a computing device and includes one or more of a magnetic medium, (e.g., a floppy diskette, a hard disk drive), an optical medium (e.g., a DVD), a semiconductor medium (e.g., a solid state hard drive), or any suitable combination thereof. The computer readable storage medium includes instructions for a computing device that direct the computing device to perform the model training optimization method.

[0365] Finally, it should be noted that the above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application; although the above-mentioned embodiments of the present application are described in detail, those skilled in the art should understand that the technical solutions recorded in the above-mentioned embodiments can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

[0366] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0367] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0368] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0369] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0370] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the application. Accordingly, it is intended that the present application embrace all such modifications and changes as fall within the scope of the claims and their equivalents.

Claims

1. A model training optimization method, characterized in that, The method comprises: determining R training devices, S batches and a number W1 of batches capable of being continuously executed in each training device of a neural network model performing pipeline parallel training, wherein R is less than S, and R, S and W1 are all greater than 0; determining S forward tasks and S reverse tasks when training the S batches on each training device, a forward calculation time length of each forward task in the S forward tasks, a reverse calculation time length of each reverse task in the S reverse tasks, and a communication time length of a forward task or a reverse task corresponding to a same batch between two training devices; determining a first execution order of the S forward tasks and the S reverse tasks on each training device in the R training devices according to the forward calculation time length of each forward task executed by each computing device, the reverse calculation time length of each reverse task, the communication time length of a forward task or a reverse task corresponding to a same batch between two training devices, and the number W1 of batches capable of being continuously executed in each training device, wherein the first execution order is an execution order in which a training time length of the R training devices training the neural network model using the S batches satisfies a first condition, and the first execution order comprises at least one group of W2 continuously executed forward tasks, wherein W2 is greater than 0 and less than or equal to W1.

2. The method of claim 1, wherein, The determining a first execution order of the S forward tasks and the S reverse tasks on each training device in the R training devices according to the forward calculation time length of each forward task executed by each computing device, the reverse calculation time length of each reverse task, the communication time length of a forward task or a reverse task corresponding to a same batch between two training devices, and the number W1 of batches capable of being continuously executed in each training device comprises: determining a target function and a constraint condition of the target function according to the forward calculation time length of each forward task executed by each computing device, the reverse calculation time length of each reverse task, the communication time length of a forward task or a reverse task corresponding to a same batch between two training devices, and the number W1 of batches capable of being continuously executed in each training device, wherein the target function is a training time length of the neural network model satisfying the first condition; solving the target function according to the constraint condition by using a linear solver to obtain the first execution order of the S forward tasks and the S reverse tasks on each training device.

3. The method of claim 2, wherein, The constraint condition comprises at least one of: an ending time of an s-th forward task on an r-th training device is less than a starting time of the s-th forward task on an r+1-th training device, r is an integer taking values in [1, R], and s is an integer taking values in [1, S]; or an ending time of the s-th forward task on the r-th training device is less than or equal to a starting time of an s+1-th forward task on the r-th training device; or The end time of the s-th backward task on the r+1-th training device is less than the start time of the s-th backward task on the r-th training device; or, The end time of the s-th backward task on the r-th training device is less than or equal to the start time of the s+1-th backward task on the r-th training device; or, The end time of the s-th forward task on the r-th training device is less than or equal to the start time of the s-th backward task on the r-th training device; or, In the case that the s1-th forward task is executed before the s2-th backward task, the end time of the s1-th forward task on the r-th training device is less than or equal to the start time of the s2-th backward task; in the case that the s2-th backward task is executed before the s1-th forward task, the end time of the s2-th backward task on the r-th training device is less than or equal to the start time of the s1-th forward task, s1 is an integer taking values from [1, S], s2 is an integer taking values from [1, S], and s1 is not equal to s2; or, The number of batches continuously executed in the r-th training device is less than or equal to W1.

4. The method according to any one of claims 1 to 3, characterized in that, The first condition is that the training time length of the R training devices in training the neural network model using the S batches according to the first execution order is the minimum.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: establishing a forward communication domain for forward tasks on the R training devices; establishing a backward communication domain for backward tasks on the R training devices.

6. The method of claim 5, wherein, The establishment of the forward communication domain for forward tasks on the R training devices comprises: for the r-th training device in the R training devices, the following steps are performed: by executing the s-th forward task through the r-th training device, a first feature vector corresponding to the s-th forward task is obtained, the first feature vector being a forward calculation feature vector corresponding to the last network layer allocated to the r-th training device; a forward communication domain between the r-th training device and the r+1-th training device is established; the forward communication domain between the r-th training device and the r+1-th training device is used to send the first feature vector corresponding to the s-th forward task to the r+1-th training device.

7. The method of claim 5, wherein, The establishment of the backward communication domain for backward tasks on the R training devices comprises: for the r-th training device in the R training devices, the following steps are performed: by executing the s-th backward task through the r-th training device, a second feature vector corresponding to the s-th backward task is obtained, the second feature vector being a backward calculation feature vector corresponding to the first network layer allocated to the r-th training device; a backward communication domain between the r-th training device and the r-1-th training device is established; the backward communication domain between the r-th training device and the r-1-th training device is used to send the second feature vector corresponding to the s-th backward task to the r-1-th training device.

8. A model training system, comprising: comprise: a computing device and a training device, the computing device is configured to: determine R training devices, S batches, and a number W1 of batches that can be continuously executed in each training device of a neural network model that is trained in pipeline parallelism, wherein R is less than S, and R, S, and W1 are greater than 0; determine S forward tasks and S backward tasks when the S batches are trained on each training device, a forward computation duration of each of the S forward tasks, a backward computation duration of each of the S backward tasks, and a communication duration of a forward task or a backward task corresponding to a same batch of data between two training devices; determine a first execution order of the S forward tasks and the S backward tasks on each of the R training devices according to the forward computation duration of each of the forward tasks executed by each computing device, the backward computation duration of each of the backward tasks, the communication duration of the forward task or the backward task corresponding to the same batch of data between two training devices, and the number W1 of batches that can be continuously executed in each training device, wherein the first execution order is an execution order in which a training duration of the R training devices training the neural network model using the S batches satisfies a first condition, and the first execution order includes at least one group of W2 continuous forward tasks, W2 is greater than 0 and less than or equal to W1; the training device is configured to train the neural network model using the S batches according to the first execution order. 9.The model training system of claim 8, wherein, The computing device is specifically configured to: determine an objective function and a constraint condition of the objective function according to the forward computation duration of each of the forward tasks executed by each computing device, the backward computation duration of each of the backward tasks, the communication duration of the forward task or the backward task corresponding to the same batch of data between two training devices, and the number W1 of batches that can be continuously executed in each training device, wherein the objective function is a training duration of the neural network model satisfying the first condition; solve the objective function according to the constraint condition using a linear solver to obtain the first execution order of the S forward tasks and the S backward tasks on each of the training devices. 10.The model training system of claim 8, wherein, The computing device is specifically configured to: establish a forward communication domain for the forward tasks on the R training devices; establish a backward communication domain for the backward tasks on the R training devices.

11. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The instructions, when executed by a computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-7.

Images