A method, apparatus, storage medium, and electronic device for task execution

Abstract

This specification discloses a method, apparatus, storage medium, and electronic device for task execution. The method for task execution includes: obtaining model data of a target model, for each computing unit, determining the maximum number of times of backpropagation allowed for each micro-training batch before each time node of the computing unit, and determining the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node, determining the time node for the computing unit to perform backpropagation for each micro-training batch among the time nodes at least before the forward propagation node as the backpropagation node, determining a training strategy for the target model according to the forward propagation node corresponding to each computing unit and the backpropagation node corresponding to each computing unit, and performing a task execution task for the target model based on the training strategy.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and particularly to a method, apparatus, storage medium, and electronic device for task execution. Background Art

[0002] With the development of deep learning technology, some ultra-large-scale models have emerged. The number of parameters required for these models to be calculated has reached hundreds of billions or even trillions. Therefore, some pipelined parallel algorithms that can be used in a multi-machine and multi-GPU cluster environment have been proposed to parallelly execute the operations involved in the model through multiple computing units (such as Graphics Processing Units (GPUs)).

[0003] In the pipelined parallel algorithm, a training batch is usually divided into multiple micro-training batches (Microbatch), and then the forward propagation and backward propagation of multiple micro-batches are completed through different computing units.

[0004] However, currently, during the task execution process using the pipelined parallel algorithm, the time and order settings for the forward propagation process and backward propagation process of each micro-batch are not reasonable, resulting in the activation values stored in the cache of the computing unit not being released in a timely manner during task execution, with a low model convergence rate, which further affects the training efficiency of the model.

[0005] Therefore, how to effectively release the activation values generated during the task execution process, improve the model convergence rate and training efficiency, is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides a method, apparatus, storage medium, and electronic device for task execution to partially solve the above problems existing in the prior art.

[0007] This specification adopts the following technical solutions:

[0008] This specification provides a method for task execution, including:

[0009] Obtaining model data of a target model;

[0010] According to the model data, splitting the target model to obtain each sub-model, and each sub-model includes some network layers of the target model;

[0011] Deploying each sub-model in different computing units;

[0012] For each computing unit, determine the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node, and determine the time node corresponding to the maximum number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch;

[0013] Among the time nodes at least before the forward propagation node, determine the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch;

[0014] According to the forward propagation node corresponding to each computing unit and the backpropagation node corresponding to each computing unit, determine the training strategy for the target model, and based on the training strategy, execute the task for the target model.

[0015] Optionally, determining the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node specifically includes:

[0016] Divide the current training batch of the target model into at least two micro-batches;

[0017] Determine the maximum number of times the computing unit is allowed to perform backpropagation for the at least two micro-training batches before each time node.

[0018] Optionally, for each time node, when the computing unit does not perform forward propagation for any one of the micro-training batches at this time node, the remaining training duration after this time node is sufficient to complete the forward propagation and backpropagation of this micro-training batch and the forward propagation and backpropagation of other micro-training batches after this micro-training batch.

[0019] Optionally, for each computing unit, determine the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node, specifically including:

[0020] For each micro-training batch and each time node, determine the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node according to at least one of the maximum number of times the computing unit performs backpropagation for the previous micro-training batch at the previous time node, the maximum number of times the computing unit performs backpropagation for the previous micro-training batch at the two previous time nodes, and the maximum number of times the computing unit performs backpropagation for this micro-training batch at the previous time node.

[0021] Optionally, for each micro-training batch and each time node, based on at least one of the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the two previous time nodes, and the maximum number of times of performing backpropagation for this micro-training batch by this computing unit at the previous time node, determine the maximum number of times of allowing backpropagation for each micro-training batch before each time node by this computing unit, specifically including:

[0022] If this computing unit is the last computing unit to perform computing, then for each time node, if the forward propagation for this micro-training batch is completed before this time node, based on the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the previous time node, and the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the two previous time nodes, determine the maximum number of times of allowing backpropagation for this micro-training batch by this computing unit before this time node.

[0023] Optionally, for each micro-training batch and each time node, based on at least one of the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the two previous time nodes, and the maximum number of times of performing backpropagation for this micro-training batch by this computing unit at the previous time node, determine the maximum number of times of allowing backpropagation for each micro-training batch before each time node by this computing unit, specifically including:

[0024] If this computing unit is not the last computing unit to perform computing, then for each time node, if the next computing unit completes the backpropagation for this micro-training batch before the previous time node, based on the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the previous time node, and the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the two previous time nodes, determine the maximum number of times of allowing backpropagation for this micro-training batch by this computing unit before this time node.

[0025] Optionally, for each micro-training batch and each time node, based on at least one of the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of performing backpropagation for the previous micro-training batch by this computing unit at the two previous time nodes, and the maximum number of times of performing backpropagation for this micro-training batch by this computing unit at the previous time node, determine the maximum number of times of allowing backpropagation for each micro-training batch before each time node by this computing unit, specifically including:

[0026] For each time node, when the computing unit is the last computing unit to perform computing and the forward propagation for the micro-training batch has not been completed before this time node, and when the computing unit is not the last computing unit to perform computing and the next computing unit has not completed the backward propagation for the training batch before the previous time node, determine the maximum number of times the backward propagation for the micro-training batch is allowed to be performed by this computing unit before this time node according to the maximum number of times of backward propagation for the micro-training batch performed by this computing unit at the previous time node of this time node.

[0027] Optionally, determining the time node corresponding to the maximum value of the number of backward propagations as the forward propagation node for the computing unit to perform the forward propagation for each micro-training batch specifically includes:

[0028] Determine the earliest time node corresponding to the maximum value of the number of backward propagations for each micro-training batch as the forward propagation node.

[0029] Optionally, among the time nodes at least before the forward propagation node, determine the time nodes for the computing unit to perform the backward propagation for each micro-training batch as the backward propagation nodes for the computing unit to perform the backward propagation for each micro-training batch, specifically including:

[0030] If the computing unit is the last computing unit to perform computing, then for each time node before the forward propagation node, if the computing unit has performed the forward propagation for any one of the micro-training batches before this time node, determine that the computing unit performs the backward propagation for this micro-training batch at this time node and use this time node as the backward propagation node.

[0031] Optionally, the method further includes:

[0032] Update the state of the computing unit at this time node to the state of having completed the backward propagation for the micro-training batch.

[0033] Optionally, among the time nodes at least before the forward propagation node, determine the time nodes for the computing unit to perform the backward propagation for each micro-training batch as the backward propagation nodes for the computing unit to perform the backward propagation for each micro-training batch, specifically including:

[0034] If the computing unit is not the last computing unit to perform computations, then for each time node located before the forward propagation node, if the next computing unit performs backpropagation for any one of the micro-training batches at the previous time node, it is determined that the computing unit performs backpropagation for the micro-training batch at this time node, and this time node is used as the backpropagation node.

[0035] Optionally, among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch are determined as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including:

[0036] For each time node located before the forward propagation node, when the computing unit is the last computing unit to perform computations and the computing unit has not performed forward propagation for one of the micro-training batches before this time node, and when the computing unit is not the last computing unit to perform computations and the next computing unit performs backpropagation for the micro-training batch at the previous time node, it is determined that no backpropagation is performed at this time node.

[0037] Optionally, for each computing unit, the maximum number of times that the computing unit is allowed to perform backpropagation for each micro-training batch before each time node is determined, and the time node corresponding to the maximum value of the number of backpropagation times is determined as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch, specifically including:

[0038] The forward propagation node is determined through a dynamic programming (DP) algorithm optimizer;

[0039] Among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch are determined as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including:

[0040] The backpropagation node is determined through a greedy algorithm optimizer.

[0041] This specification provides a task execution device, including:

[0042] An acquisition module that acquires model data of a target model;

[0043] A splitting module that splits the target model according to the model data to obtain each sub-model, and each sub-model includes some network layers of the target model;

[0044] A deployment module that deploys each sub-model in different computing units;

[0045] A first determination module, for each computing unit, determines the maximum number of times of backpropagation allowed for each micro-training batch before each time node, and determines the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch;

[0046] A second determination module determines, among the time nodes at least before the forward propagation node, the time nodes for the computing unit to perform backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch;

[0047] An execution module determines a training strategy for the target model according to the forward propagation nodes corresponding to each computing unit and the backpropagation nodes corresponding to each computing unit, and executes a task execution task for the target model based on the training strategy.

[0048] This specification provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method for task execution described above is implemented.

[0049] This specification provides an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for task execution described above is implemented.

[0050] At least one of the technical solutions adopted in this specification can achieve the following beneficial effects:

[0051] In the method for task execution provided in this specification, model data of a target model is obtained. For each computing unit, the maximum number of times of backpropagation allowed for each micro-training batch before each time node is determined, and the time node corresponding to the maximum value of the number of backpropagation times is determined as the forward propagation node. Among the time nodes at least before the forward propagation node, the time nodes for the computing unit to perform backpropagation for each micro-training batch are determined as the backpropagation nodes. A training strategy for the target model is determined according to the forward propagation nodes corresponding to each computing unit and the backpropagation nodes corresponding to each computing unit, and a task execution task for the target model is executed based on the training strategy.

[0052] As can be seen from the above method, during the execution of the task, this solution can determine the time node corresponding to the maximum number of backpropagation times for each micro-training batch as its corresponding forward propagation node. In this way, as many backpropagations as possible can be performed before each micro-batch forward propagation, advancing the time of partial micro-batch backpropagation. This can not only release the activation values stored in the cache earlier, but also increase the number of model parameter updates as the number of backpropagations increases. Then, when performing forward propagation again, the convergence rate of the model will also increase, further improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of this specification and form a part of this specification. The schematic embodiments and descriptions thereof in this specification are used to explain this specification and do not constitute an improper limitation to this specification. In the drawings:

[0054] Figure 1 is a schematic flowchart of a method for task execution provided in this specification;

[0055] Figure 2 is a schematic diagram of the correspondence between each computing unit and each network layer provided in this specification;

[0056] Figure 3 is a schematic flowchart of determining a training strategy based on a DP algorithm optimizer and a greedy algorithm optimizer provided in this specification;

[0057] Figure 4 is a schematic diagram of the pipeline parallel computing process of a target model provided in this specification;

[0058] Figure 5 is a schematic diagram of a device for task execution provided in this specification;

[0059] Figure 6 is provided in this specification corresponding to Figure 1 schematic diagram of an electronic device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0061] The following will detail the technical solutions provided in each embodiment of this specification in conjunction with the drawings.

[0062] Figure 1 This is a flowchart showing a method for task execution provided in this specification, including the following steps:

[0063] S101: Obtain the model data of the target model.

[0064] S102: Split the target model according to the model data to obtain each sub-model, and each sub-model contains some network layers of the target model.

[0065] S103: Deploy each sub-model in different computing units.

[0066] Obtain the model data of the target model, and deploy each network layer of the target model in different computing units.

[0067] Current pipeline parallel algorithms usually include, for example, Gpipe, Pipedream, virtual Pipeline, etc. However, these pipeline parallel algorithms all have problems of high cache occupancy and slow model convergence. Take the synchronous parallel method of Gpipe as an example. Since the progress arrangement of the backward propagation part is after the forward propagation part, the activation values cannot be released in time, resulting in high cache occupancy. Take the asynchronous parallel methods of Pipedream and virtual Pipeline as an example. Before the backward propagation of the previous micro-batch is completed and the weights are not updated, the neural network layer has already started the forward propagation of the next micro-batch. This operation leads to low model convergence efficiency.

[0068] Based on this, this specification provides a method for task execution, which delays the execution time of the forward propagation as much as possible without affecting the final training completion time, and advances the execution time of part of the backward propagation as much as possible, so as to improve the model convergence rate and release the cache occupied by the activation values in the forward propagation process in time.

[0069] In this specification, the execution entity for implementing the method for task execution may refer to a specified device such as a server. For the sake of convenience of description, this specification only takes the server as the execution entity as an example to illustrate a method for task execution provided in this specification.

[0070] Among them, after receiving the training request of the target model, the server can first obtain the model data of the target model, and then determine each network layer included in the target model.

[0071] Subsequently, the server may deploy each network layer of the target model in different computing units. In this specification, the computing unit may be a GPU. Of course, it may also be other computing devices such as a Central Processing Unit (CPU), a Tensor Processing Unit (TPU), or a Neural-Network Processing Unit (NPU). This specification does not make specific limitations in this regard.

[0072] Specifically, the server may split the target model according to the model data to obtain each sub-model. Each sub-model contains some network layers of the target model, and then deploy each sub-model in different computing units.

[0073] Among them, the server may divide the target model into a layers. When the total number of computing units is n, the server may set every a / n network layers of the target model from the input layer to the output layer as a group. Among them, the total number of network layers a must be divisible by the total number of computing units n.

[0074] Subsequently, the server may allocate each group of network layers to each computing unit. That is, the first group (including the first layer to the a / n layer) is allocated to computing unit 1, the second group (including the (a / n)+1 layer to the 2a / n layer) is allocated to computing unit 2, ……, the nth group (including the ((n - 1)×(a / n)+1 layer to the a layer) is allocated to computing unit n. For ease of understanding, this specification provides a schematic diagram of the correspondence between each computing unit and each network layer, as Figure 2 shown.

[0075] Figure 2 This is a schematic diagram of the correspondence between each computing unit and each network layer provided in this specification.

[0076] Among them, in addition to the input layer and the output layer, the target model also contains 16 other network layers (Layers), which are respectively allocated to different GPUs for computing.

[0077] S104: For each computing unit, determine the maximum number of times that the computing unit is allowed to perform backpropagation for each micro-training batch before each time node, and determine the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch.

[0078] In order to postpone the forward propagation as much as possible and advance the backward propagation, in this specification, this problem can be equivalently considered as how many backward propagations can be performed at most when each computing unit performs the forward propagation of each micro-training batch at each time node. Thus, the time node when the number of backward propagations reaches the maximum value is taken as the forward propagation node, so as to perform as many backward propagations as possible before this forward propagation node and postpone the time of forward propagation.

[0079] In practical applications, each forward propagation of a computing unit usually occupies one time node, and each backward propagation usually occupies two time nodes. The specific time interval between each time node can be set according to the actual situation, and this specification does not make specific limitations on this.

[0080] In addition, in this specification, the server can divide the current training batch into multiple micro-training batches (Microbatch), and then determine the forward propagation nodes for each computing unit to perform the forward propagation of each micro-training batch.

[0081] Specifically, for any time node j, if computing unit i does not perform the forward propagation of micro-training batch k at time node j, it is necessary to ensure that: the remaining time (m - j) is sufficient to complete the forward propagation and backward propagation of micro-training batch k, and the remaining time (m - j) is sufficient for the remaining z - k micro-training batches to complete the forward propagation and backward propagation. Here, m is the total training duration of the current training batch.

[0082] When computing unit i is the last computing unit to perform calculations, only the backward propagation of micro-training batch k needs to be completed, which takes 2n. However, if computing unit i is not the last computing unit to perform calculations, it is necessary to ensure that the remaining forward propagation and backward propagation of micro-training batch k are completed, which takes (n - i) + 2n. In addition, if computing unit i does not perform the forward propagation of micro-training batch k at time node j, it is necessary to ensure that the remaining time (m - j) is sufficient to complete the forward propagation and backward propagation of micro-training batch k, and the remaining time (m - j) is sufficient to complete the forward propagation and backward propagation of the remaining z - k micro-training batches, which takes (z - k) * (1 + 2), where z is the total number of micro-training batches.

[0083] That is to say, the following relationships exist for time node j:

[0084] m - j >= 2n + (z - k) * (1 + 2), If i = n

[0085] m - j >= (n - i) + 2n + (z - k) * (1 + 2), If i ≠ n

[0086] Based on this, for any time node j, its corresponding constraint relationship can be expressed as:

[0087]

[0088] For any computing unit i, any micro-training batch k, and any time node j, the server can determine the maximum number of times of backpropagation allowed for the micro-training batch for the computing unit i before each time node according to at least one of the maximum number of times of backpropagation for the previous micro-training batch (k - 1) executed by the computing unit i at the previous time node (j - 1) of the time node j, the maximum number of times of backpropagation for the previous micro-training batch (k - 1) executed by the computing unit i at the two previous time nodes (j - 2), and the maximum number of times of backpropagation for the micro-training batch k executed by the computing unit i at the previous time node (j - 1) of the time node j. Among them, the maximum number of times of backpropagation allowed for the computing unit i to execute for the micro-training batch k before the time node j can be expressed as DP(i, j, k).

[0089] In practical applications, for the forward propagation of each batch, it can only evolve from the forward propagation of the previous micro-training batch (k - 1). And DP(i, j, k) can increase by at most 1 or remain unchanged from the previous state at a time. If DP(i, j, k) increases by 1, it can only be that the computing unit i performed an additional backpropagation at the time node j - 2. If DP(i, j, k) remains unchanged, it means that no backpropagation was performed after the previous micro-training batch (k - 1), and the forward propagation of the micro-training batch k was directly performed, so the number of backpropagations does not increase.

[0090] Specifically, the server can first set the base value of DP(i, j, k). In this specification, it can be set that DP(i, 1…3n - 2i, 1) = 0 and DP(i, 3n - 2i, 1) = 1. That is to say, the maximum number of backpropagations for the first micro-training batch (starting micro-training batch) for the device i before the time node 3n - 2i is 0, that is, only forward propagation is performed, and the first backpropagation is performed at the time node 3n - 2i.

[0091] If the computing unit i is the last computing unit to perform the calculation, for any time node j, if the forward propagation for the micro-training batch k is completed before this time node j, then according to the maximum number of backpropagations for the previous micro-training batch (k - 1) executed by the computing unit i at the previous time node (j - 1), and the maximum number of backpropagations for the previous micro-training batch (k - 1) executed by the computing unit i at the two previous time nodes (j - 2), determine the maximum number of backpropagations allowed for the computing unit i to perform for the micro-training batch k before the time node j.

[0092] In this specification, the server may take the maximum number of times that computing unit i performs backpropagation for the previous micro-training batch (k - 1) at the previous time node (j - 1), and the maximum number of times that computing unit i performs backpropagation for the previous micro-training batch (k - 1) at the two previous time nodes (j - 2), and take the maximum value as the maximum number of times that computing unit i is allowed to perform backpropagation for micro-training batch k before time node j. The maximum number of times that computing unit i is allowed to perform backpropagation for micro-training batch k before time node j, DP(i, j, k), can be expressed as:

[0093] DP(i, j, k) = max(DP(i, j - 2, k - 1) + 1, DP(i, j - 1, k - 1))

[0094] where the time node j satisfies j <= m - 2n - 3(z - k).

[0095] If computing unit i is not the last computing unit to perform calculations, then for any time node j, when the next computing unit (i + 1) completes backpropagation for micro-training batch (k) before the previous time node (j - 1), then according to the maximum number of times that computing unit i performs backpropagation for the previous micro-training batch (k - 1) at the previous time node (j - 1), and the maximum number of times that this computing unit performs backpropagation for the previous micro-training batch (k - 1) at the two previous time nodes (j - 2), determine the maximum number of times that computing unit i is allowed to perform backpropagation for micro-training batch k before time node j.

[0096] In this specification, the server may take the maximum number of times that computing unit i performs backpropagation for the previous micro-training batch (k - 1) at the previous time node (j - 1), and the maximum number of times that computing unit i performs backpropagation for the previous micro-training batch (k - 1) at the two previous time nodes (j - 2), and take the maximum value as the maximum number of times that computing unit i is allowed to perform backpropagation for micro-training batch k before time node j. The maximum number of times that computing unit i is allowed to perform backpropagation for micro-training batch k before time node j, DP(i, j, k), can be expressed as:

[0097] DP(i, j, k) = max(DP(i, j - 2, k - 1) + 1, DP(i, j - 1, k - 1))

[0098] where the time node j satisfies j <= i + m - 3(n + z - k).

[0099] If the above two cases are not satisfied, that is, for any time node j, when the computing unit i is the last computing unit to perform calculations and the forward propagation for the micro-training batch k has not been completed before the time node j, and when the computing unit i is not the last computing unit to perform calculations and the next computing unit (k + 1) has not completed the backward propagation for the training batch k before the previous time node (j - 1), the server can determine the maximum number of times the computing unit i is allowed to perform backward propagation for the micro-training batch before the time node j according to the maximum number of backward propagation times for the micro-training batch at the previous time node (j - 1) of the computing unit i at the time node j. The maximum number of times the computing unit i is allowed to perform backward propagation for the micro-training batch before the time node j, DP(i, j, k), can be expressed as:

[0100] DP(i, j, k) = DP(i, j - 1, k)

[0101] Based on this, the server can obtain the dynamic transition equation, which can be expressed as:

[0102]

[0103] The server can determine the maximum number of times each computing unit is allowed to perform backward propagation for each micro-training batch before each time node based on this dynamic transition equation.

[0104] Furthermore, the server can determine the time node corresponding to the maximum value of the backward propagation times of each micro-training batch based on the above dynamic transition equation as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch.

[0105] Specifically, the server can traverse each computing unit and each time node according to the above dynamic transition equation to determine the earliest time node j* when DP(i, j, k) is the maximum value as the forward propagation node when the computing unit i performs forward propagation for the micro-training batch k, and then obtain the two-dimensional array DP_res(i, j*) = k, indicating that the device i performs forward propagation for the micro-batch k when DP(i, j, k) is the maximum value and the time node is j*.

[0106] The server can determine the forward propagation strategy of the computing unit according to the forward propagation node for the computing unit to perform forward propagation for each micro-training batch.

[0107] It should be noted that for the forward propagation node corresponding to each micro-training batch, the server has completed the forward propagation for the previous micro-training batch before this propagation node. That is, the forward propagation node of the previous micro-training batch needs to be before the forward propagation node corresponding to the current micro-training batch.

[0108] In this specification, the server can also use a Dynamic Programming (DP) optimizer to determine the above dynamic programming equation based on information such as the model structure of the target model, the dataset parameters (such as mean, variance, etc.) of the training dataset, and the environmental parameters of the computing units, and further determine the forward propagation strategy of each computing unit according to the forward propagation nodes of each computing unit for each micro-training batch during forward propagation.

[0109] S105: Determine, among the time nodes that are at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes at which the computing unit performs backpropagation for each micro-training batch.

[0110] After determining the forward propagation nodes, the server can determine the time nodes at which each computing unit performs backpropagation for each micro-batch among the other time nodes that are at least before the determined forward propagation nodes.

[0111] In other words, the server can determine as many backpropagation nodes as possible before the forward propagation nodes, so as to perform as much backpropagation as possible before each forward propagation. Of course, there are also some micro-training batches for which the backpropagation is after the forward propagation nodes.

[0112] Specifically, for any computing unit i, if the computing unit i is the last computing unit to perform calculations, then for any time node j, if the computing unit i performs forward propagation for any micro-training batch k before the time node j, then determine that the computing unit i performs backpropagation for the micro-training batch k at the time node j, and use the time node j as the backpropagation node at which the computing device i performs backpropagation for the training batch k, denoted as D_e(i,j)=k, and update the state of the computing unit i at the time node j to the state of having completed backpropagation for the micro-training batch k, denoted as D_b(i)=k.

[0113] If the computing unit i is not the last computing unit to perform calculations, then for any time node j, if the next computing unit (i + 1) performs backpropagation for any micro-training batch (k) at the previous time node (j - 1), then determine that the computing unit i performs backpropagation for the micro-training batch k at the time node j, and use the time node j as the backpropagation node at which the computing device i performs backpropagation for the micro-training batch k.

[0114] For other cases other than the above, that is, the computing unit i is the last computing unit to perform calculations and the computing unit i has not performed forward propagation for the micro-training batch k before the time node j, and the computing unit i is not the last computing unit to perform calculations and the next computing unit (k + 1) performs backward propagation for the micro-training batch k at the previous time node (j - 1), it is determined that backward propagation is not performed at the time node j.

[0115] Based on this, the server can obtain the update equation of the backward propagation node, and this update equation can be expressed as:

[0116]

[0117] The server can determine whether the computing unit i performs backward propagation for the micro-training batch k at the time node j according to the above update equation, and then determine the backward propagation nodes for each computing unit to perform backward propagation for each micro-training batch.

[0118] In this specification, the server can determine the backward propagation nodes for each computing unit to perform backward propagation for each micro-training batch through a greedy algorithm optimizer according to information such as the model structure information of the target model, dataset parameters, and environmental parameters of the computing unit, as well as the output of the DP algorithm optimizer. For the convenience of understanding, this specification provides a flow schematic diagram for determining the training strategy based on the DP algorithm optimizer and the greedy algorithm optimizer, as Figure 3 shown.

[0119] Figure 3 It is a flow schematic diagram for determining the training strategy based on the DP algorithm optimizer and the greedy algorithm optimizer provided in this specification.

[0120] Among them, the server can obtain the model structure information, dataset parameters, and environmental parameters of the computing unit, input them into the DP algorithm optimizer, output the forward propagation strategy of each computing unit, and then input the output of the DP algorithm optimizer, model structure information, dataset parameters, and environmental parameters of the computing unit into the greedy algorithm optimizer to determine the backward propagation strategy of each computing unit, and then determine the training strategy of the target model based on the forward propagation strategy and the backward propagation strategy, and train the target model based on this training strategy.

[0121] S106: Determine the training strategy for the target model according to the forward propagation nodes corresponding to each computing unit and the backward propagation nodes corresponding to each computing unit, and perform the task execution task for the target model based on the training strategy.

[0122] After determining the forward propagation node at which each computing unit performs forward propagation for each micro-training batch, and the back-propagation node at which each computing unit performs back-propagation for each micro-training batch, the server can determine, based on each forward propagation node and each back-propagation node, at which time node each computing unit performs forward propagation for which micro-training batch, and at which computing node each computing unit performs back-propagation for which micro-training batch, thereby determining the training strategy during task execution.

[0123] The server can then follow the training strategy and, based on the training samples of each micro-training batch, call different computing units to complete the forward propagation and backpropagation of each micro-training batch, thereby executing the training task of the current batch for the target model.

[0124] For ease of understanding, this specification provides a schematic diagram of the pipeline parallel computing process of a target model, such as Figure 4 shown.

[0125] Figure 4 This is a schematic diagram of the pipeline parallel computing process of a target model provided in this specification.

[0126] Among them, each smallest rectangular box is a time node, the white boxes with numbers (i.e. 1L~4L, 1R~4R) are forward propagation, and each forward propagation occupies one time node, and the black boxes with numbers (i.e. 1L~4L, 1R~4R) are reverse propagation, and each reverse propagation occupies two time nodes.

[0127] It can be seen from the above method that during the execution of the task, this scheme can determine the time node corresponding to the maximum number of backpropagation times of each micro-training batch as its corresponding forward propagation node. In this way, as much backpropagation as possible can be performed before the forward propagation of each micro-batch, and the time when most backpropagation is performed can be advanced. This not only enables the activation values stored in the cache to be released in advance, but also the more times the backpropagation is performed, the more times the model parameters are updated. The convergence rate of the model will also increase when the forward propagation is performed later, further improving the training efficiency of the model.

[0128] The above are one or more methods for implementing task execution in this specification. Based on the same idea, this specification also provides corresponding task execution devices, such as Figure 5 shown.

[0129] Figure 5 A schematic diagram of a device for performing a task provided in this specification, including:

[0130] An acquisition module 501 is used to acquire model data of a target model;

[0131] The splitting module 502 is used to split the target model according to the model data to obtain each sub-model, and each sub-model includes some network layers of the target model;

[0132] The deployment module 503 is used to deploy each sub-model in different computing units;

[0133] The first determination module 504 is used to, for each computing unit, determine the maximum number of times of backpropagation allowed for each micro-training batch before each time node, and determine the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the forward propagation of each micro-training batch by this computing unit;

[0134] The second determination module 505 is used to determine, among the time nodes at least before the forward propagation node, the time nodes for the backpropagation of each micro-training batch by this computing unit as the backpropagation nodes for the backpropagation of each micro-training batch by this computing unit;

[0135] The execution module 506 is used to determine the training strategy for the target model according to the forward propagation node corresponding to each computing unit and the backpropagation node corresponding to each computing unit, and execute the task execution task for the target model based on the training strategy.

[0136] Optionally, the splitting module 502 is specifically used to divide the current training batch of the target model into at least two micro-batches; determine the maximum number of times of backpropagation allowed for the at least two micro-training batches by this computing unit before each time node.

[0137] Optionally, for each time node, when this computing unit does not perform the forward propagation of any one of the micro-training batches at this time node, the remaining training duration after this time node is sufficient to complete the forward propagation and backpropagation of this micro-training batch and the forward propagation and backpropagation of other micro-training batches after this micro-training batch.

[0138] Optionally, the first determination module 504 is specifically used to, for each micro-training batch and each time node, determine the maximum number of times of backpropagation allowed for each micro-training batch by this computing unit before each time node according to at least one of the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous two time nodes, and the maximum number of times of backpropagation of this micro-training batch by this computing unit at the previous time node.

[0139] Optionally, the first determination module 504 is specifically configured to, if the computing unit is the last computing unit to perform computing, then for each time node, if the forward propagation for the micro-training batch is completed before the time node, determine the maximum number of times the computing unit is allowed to perform the backward propagation for the micro-training batch before the time node according to the maximum number of times the computing unit performs the backward propagation for the previous micro-training batch at the previous time node and the maximum number of times the computing unit performs the backward propagation for the previous micro-training batch at the two previous time nodes.

[0140] Optionally, the first determination module 504 is specifically configured to, if the computing unit is not the last computing unit to perform computing, then for each time node, if the next computing unit completes the backward propagation for the micro-training batch before the previous time node, determine the maximum number of times the computing unit is allowed to perform the backward propagation for the micro-training batch before the time node according to the maximum number of times the computing unit performs the backward propagation for the previous micro-training batch at the previous time node and the maximum number of times the computing unit performs the backward propagation for the previous micro-training batch at the two previous time nodes.

[0141] Optionally, the first determination module 504 is specifically configured to, for each time node, when the computing unit is the last computing unit to perform computing and the forward propagation for the micro-training batch is not completed before the time node, and when the computing unit is not the last computing unit to perform computing and the next computing unit does not complete the backward propagation for the training batch before the previous time node, determine the maximum number of times the computing unit is allowed to perform the backward propagation for the micro-training batch before the time node according to the maximum number of times the computing unit performs the maximum backward propagation for the micro-training batch at the previous time node of the time node.

[0142] Optionally, the first determination module 504 is specifically configured to determine the earliest time node corresponding to the maximum value of the number of backward propagations of each micro-training batch as the forward propagation node.

[0143] Optionally, the second determination module 505 is specifically configured to, if the computing unit is the last computing unit to perform computing, then for each time node before the forward propagation node, if the computing unit performs the forward propagation for any one of the micro-training batches before the time node, determine that the computing unit performs the backward propagation for the micro-training batch at the time node and use the time node as the backward propagation node.

[0144] Optionally, the second determination module 505 is further configured to update the state of the computing unit at the time node to the state of having completed the backward propagation for the micro-training batch.

[0145] Optionally, the second determination module 505 is specifically configured to, if the computing unit is not the last computing unit performing calculations, for each time node before the forward propagation node, if the next computing unit performs backpropagation for any one of the micro-training batches at the previous time node, determine that the computing unit performs backpropagation for the micro-training batch at the time node, and use the time node as the backpropagation node.

[0146] Optionally, the second determination module 505 is specifically configured to, for each time node before the forward propagation node, when the computing unit is the last computing unit performing calculations and the computing unit has not performed forward propagation for one of the micro-training batches before this time node, and when the computing unit is not the last computing unit performing calculations and the next computing unit performs backpropagation for the micro-training batch at the previous time node, determine that no backpropagation is performed at this time node.

[0147] Optionally, the first determination module 504 is specifically configured to determine the forward propagation node through a dynamic programming (DP) algorithm optimizer;

[0148] Optionally, the second determination module 505 is specifically configured to determine the backpropagation node through a greedy algorithm optimizer.

[0149] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the Figure 1 task execution method provided above.

[0150] This specification also provides Figure 6 a schematic structural diagram of an electronic device corresponding to Figure 1 . As Figure 6 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 task execution method described above. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0151] Improvements to a technology can be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by a user's programming of the device. A designer can program on their own to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compilers used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there is not just one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0152] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0153] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0154] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0155] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0156] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0157] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0159] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0160] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0161] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0163] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0165] Each embodiment in this specification is described in a progressive manner. For the identical or similar parts among the embodiments, they can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

[0166] The above are only the embodiments of this specification and are not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A method for task execution, characterized in that, Including: Obtain the model data of the target model; According to the model data, split the target model to obtain each sub-model, and each sub-model includes some network layers of the target model; Deploy each sub-model in different computing units; For each computing unit, determine the maximum number of times of backpropagation allowed for each micro-training batch before each time node, and determine the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the forward propagation of each micro-training batch by this computing unit; Among the time nodes at least before the forward propagation node, determine the time nodes for the backpropagation of each micro-training batch by this computing unit as the backpropagation nodes for the backpropagation of each micro-training batch by this computing unit; According to the forward propagation nodes corresponding to each computing unit and the backpropagation nodes corresponding to each computing unit, determine the training strategy for the target model, and based on the training strategy, execute the task for the target model.

2. The method according to claim 1, wherein For each computing unit, determining the maximum number of times of backpropagation allowed for each micro-training batch before each time node specifically includes: Divide the current training batch of the target model into at least two micro-batches; Determine the maximum number of times of backpropagation allowed for the at least two micro-training batches before each time node by this computing unit.

3. The method according to claim 2, characterized in that, For each time node, when this computing unit does not perform the forward propagation of any one of the micro-training batches at this time node, the remaining training duration after this time node is sufficient to complete the forward propagation and backpropagation of this micro-training batch and the forward propagation and backpropagation of other micro-training batches after this micro-training batch.

4. The method according to claim 1, wherein For each computing unit, for each computing unit, determining the maximum number of times of backpropagation allowed for each micro-training batch before each time node specifically includes: For each micro-training batch and each time node, determine the maximum number of times of backpropagation allowed for each micro-training batch before each time node by this computing unit according to at least one of the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous two time nodes, and the maximum number of times of backpropagation of this micro-training batch by this computing unit at the previous time node.

5. The method according to claim 4, wherein For each micro-training batch and each time node, determining the maximum number of times of backpropagation allowed for each micro-training batch before each time node according to at least one of the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous time node, the maximum number of times of backpropagation of the previous micro-training batch by this computing unit at the previous two time nodes, and the maximum number of times of backpropagation of this micro-training batch by this computing unit at the previous time node specifically includes: If the computing unit is the last computing unit to perform calculations, then for each time node, if the forward propagation for the micro-training batch is completed before this time node, then based on the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the previous time node, and the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the two previous time nodes, determine the maximum number of times the computing unit is allowed to perform backpropagation for the micro-training batch before this time node.

6. The method according to claim 4, characterized in that, For each micro-training batch and each time node, based on at least one of the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the previous time node, the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the two previous time nodes, and the maximum number of times the computing unit performed backpropagation for the current micro-training batch at the previous time node, determine the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node. Specifically, it includes: If the computing unit is not the last computing unit to perform calculations, then for each time node, when the next computing unit completes the backpropagation for the micro-training batch before the previous time node, then based on the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the previous time node, and the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the two previous time nodes, determine the maximum number of times the computing unit is allowed to perform backpropagation for the micro-training batch before this time node.

7. The method according to claim 4, wherein For each micro-training batch and each time node, based on at least one of the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the previous time node, the maximum number of times the computing unit performed backpropagation for the previous micro-training batch at the two previous time nodes, and the maximum number of times the computing unit performed backpropagation for the current micro-training batch at the previous time node, determine the maximum number of times the computing unit is allowed to perform backpropagation for each micro-training batch before each time node. Specifically, it includes: For each time node, when the computing unit is the last computing unit to perform calculations and the forward propagation for the micro-training batch is not completed before this time node, and when the computing unit is not the last computing unit to perform calculations and the next computing unit does not complete the backpropagation for the training batch before the previous time node, based on the maximum number of times the computing unit performed backpropagation for the micro-training batch at the previous time node of this time node, determine the maximum number of times the computing unit is allowed to perform backpropagation for the micro-training batch before this time node.

8. The method according to claim 1, wherein Determine the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch. Specifically, it includes: Determine the earliest time node corresponding to the maximum value of the number of backpropagation times for each micro-training batch as the forward propagation node.

9. The method according to claim 1, wherein Determine, among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including: If the computing unit is the last computing unit to perform computations, then for each time node before the forward propagation node, if the computing unit performs forward propagation for any one of the micro-training batches before this time node, determine that the computing unit performs backpropagation for this micro-training batch at this time node, and use this time node as the backpropagation node.

10. The method according to claim 9, characterized in that, The method further includes: Update the state of the computing unit at this time node to the state of having completed backpropagation for this micro-training batch.

11. The method according to claim 1, characterized in that, Determine, among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including: If the computing unit is not the last computing unit to perform computations, then for each time node before the forward propagation node, if the next computing unit performs backpropagation for any one of the micro-training batches at the previous time node, determine that the computing unit performs backpropagation for this micro-training batch at this time node, and use this time node as the backpropagation node.

12. The method according to claim 1, wherein Determine, among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including: For each time node before the forward propagation node, when the computing unit is the last computing unit to perform computations and the computing unit has not performed forward propagation for one of the micro-training batches before this time node, and when the computing unit is not the last computing unit to perform computations and the next computing unit performs backpropagation for this micro-training batch at the previous time node, determine that no backpropagation is performed at this time node.

13. The method according to claim 1, characterized in that, For each computing unit, determine the maximum number of times that the computing unit is allowed to perform backpropagation for each micro-training batch before each time node, and determine the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch, specifically including: Determine the forward propagation node through a dynamic programming (DP) algorithm optimizer; Determine, among the time nodes at least before the forward propagation node, the time nodes at which the computing unit performs backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch, specifically including: Determine the backpropagation node through a greedy algorithm optimizer.

14. A device for task execution, characterized in that, Including: An acquisition module that acquires the model data of the target model; A splitting module that splits the target model according to the model data to obtain each sub-model, where each sub-model includes some network layers of the target model; A deployment module that deploys each sub-model in different computing units; A first determination module that, for each computing unit, determines the maximum number of times of backpropagation allowed for each micro-training batch before each time node, and determines the time node corresponding to the maximum value of the number of backpropagation times as the forward propagation node for the computing unit to perform forward propagation for each micro-training batch; A second determination module that determines, among the time nodes at least before the forward propagation node, the time nodes for the computing unit to perform backpropagation for each micro-training batch as the backpropagation nodes for the computing unit to perform backpropagation for each micro-training batch; An execution module that determines a training strategy for the target model according to the forward propagation node corresponding to each computing unit and the backpropagation node corresponding to each computing unit, and executes a task for the target model based on the training strategy.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 13 above is implemented.

16. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 13 above is implemented.

Images