Resource-constrained large model heterogeneous training method, computer equipment and storage medium

Abstract

The invention relates to a resource-limited large model heterogeneous training method, computer equipment and a storage medium, and the method comprises the steps: determining a first activation amount according to a dependency relationship between calculated amounts corresponding to the activation amounts and / or the calculated amount and the storage amount of each activation amount in the activation amounts generated when an image processor carries out model forward propagation, the first activation amount is asynchronously transmitted to the central processing unit; loading a first activation amount to the image processor from the central processing unit in a model back propagation process; obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount, and asynchronously transmitting the gradient to the central processing unit; and determining an optimizer obtained based on gradient updating in the central processing unit, obtaining model parameters obtained by adjustment of the updated optimizer, and asynchronously transmitting the model parameters to the image processor. By adopting the method, the problems that a large model with limited video memory resources is difficult to train and the computing efficiency of a video card is low can be solved.

Description

Technical Field

[0001] The present application relates to the field of model training, and in particular to a resource-constrained large model heterogeneous training method, computer equipment, and storage medium. Background Art

[0002] In the field of deep learning, the training and inference process of large models consumes a lot of computing resources, storage space and communication bandwidth. Especially when it comes to large models with billions or even trillions of parameters, the traditional single hardware architecture is difficult to meet the training needs. Most of the current practices are to expand computing storage resources and use distributed training strategies, but these strategies usually need to rely on expensive large-scale hardware clusters, which is not feasible for resource-constrained research institutions or enterprises. It will make it difficult to complete the training of large models with video memory requirements greater than physical video memory and ensure the computing efficiency of the graphics card when video memory resources are very limited.

[0003] There is currently no effective solution to the problem that large models with limited video memory resources are difficult to train and graphics card computing efficiency is low in related technologies. Summary of the invention

[0004] Based on this, it is necessary to provide a resource-constrained large model heterogeneous training method, computer equipment and storage medium that can solve the problems of difficulty in training large models with limited video memory resources and low graphics card computing efficiency in response to the above technical problems.

[0005] In a first aspect, a resource-constrained large model heterogeneous training method is provided in this embodiment, and the method includes:

[0006] Among the activations generated when the image processor performs forward propagation of the model, a first activation is determined according to a dependency relationship between calculation amounts corresponding to each of the activations and / or a calculation amount and storage amount of each of the activations, and the first activation is asynchronously transmitted to the central processing unit;

[0007] loading the first activation from the central processing unit to the image processor during the back propagation of the model;

[0008] Obtaining a gradient of a model parameter calculated by the image processor based on the first activation amount, and asynchronously transmitting the gradient to the central processing unit;

[0009] An optimizer updated based on the gradient in the central processing unit is determined, and model parameters adjusted by the updated optimizer are obtained, and the model parameters are asynchronously transmitted to the image processor.

[0010] In some embodiments, the model includes a plurality of basic building blocks, and the step of obtaining the gradient of the model parameters calculated by the image processor based on the first activation amount and asynchronously transmitting the gradient to the central processing unit includes:

[0011] Whenever the image processor calculates the gradient of the parameter of the basic building unit based on the first activation amount, the gradient corresponding to the current basic building unit is asynchronously transmitted to the central processing unit in units of the basic building unit.

[0012] In some of the embodiments, after obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount, the method further includes:

[0013] A second activation amount used by the image processor when calculating a gradient of a model parameter based on the first activation amount is released.

[0014] In some of the embodiments, among the activations generated when the image processor performs forward propagation of the model, determining a first activation according to a dependency relationship between calculation amounts corresponding to each of the activations and / or a calculation amount and storage amount of each of the activations, and asynchronously transmitting the first activation to the central processor includes:

[0015] In the process of forward propagation of the model, the activation amount corresponding to the input of the model, the activation amount calculated based on the redundant term, and the activation amount whose calculation amount and storage amount are respectively greater than the corresponding specified value are used as the first activation amount;

[0016] The first activation amount is asynchronously transmitted to the central processing unit.

[0017] In some of the embodiments, the central processing unit stores model parameters and an optimizer of a first precision, the image processor stores model parameters of a second precision and gradients of the model parameters of the second precision, and the first precision is higher than the second precision.

[0018] In some embodiments, the asynchronously transmitting the model parameters to the image processor comprises:

[0019] replicating the model parameters based on the second precision;

[0020] The model parameters of the second precision obtained by asynchronous transmission are copied to the image processor.

[0021] In some embodiments, after transmitting the optimizer asynchronous parameters to the image processor, the method further comprises:

[0022] Determine whether the parameters of the model have been updated;

[0023] If not, repeat the following steps: asynchronously transmit the first activation amount to the central processing unit during the forward propagation of the model; load the first activation amount from the central processing unit to the image processor during the reverse propagation of the model; load the first activation amount from the central processing unit to the image processor during the reverse propagation of the model; obtain the gradient of the model parameters calculated by the image processor based on the first activation amount, and asynchronously transmit the gradient to the central processing unit; determine the optimizer updated based on the gradient in the central processing unit, and obtain the model parameters adjusted by the updated optimizer, and asynchronously transmit the model parameters to the image processor.

[0024] In some of these embodiments, the method further comprises:

[0025] Dividing the training data of the model to obtain multiple training sub-data;

[0026] The plurality of training sub-data are traversed, the traversed training sub-data are input into the model, and forward propagation and backward propagation are performed on the model based on the training sub-data set until the traversal of the plurality of training sub-data is completed.

[0027] In a second aspect, a computer device is provided in this embodiment, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the resource-constrained large model heterogeneous training method described in the first aspect is implemented.

[0028] According to a third aspect, a computer-readable storage medium is provided in this embodiment, on which a computer program is stored. When the computer program is executed by a processor, the resource-constrained large model heterogeneous training method described in the first aspect is implemented.

[0029] The above-mentioned resource-constrained large model heterogeneous training method, computer equipment and storage medium asynchronously transmit part of the activation amount generated during the forward propagation of the model to the central processing unit, load the activation amount required for the back propagation of the model on the image processor, and asynchronously transmit the gradient calculated based on the activation amount to the central processing unit, so that the central processing unit can obtain the optimizer parameters according to the gradient in real time, reducing the extra time caused by unloading, loading and recalculating the activation amount, improving the computing efficiency during large model training, and solving the problem of difficult training of large models with limited video memory resources and low computing efficiency of graphics cards. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a process flow of a resource-constrained large model heterogeneous training method in one embodiment;

[0031] Figure 2A schematic diagram of the activation amount and computation amount generated by the forward propagation of a Transformer-block in one embodiment;

[0032] Figure 3 Schematic diagram of dependency relationships in a Transformer-block in one embodiment;

[0033] Figure 4 Schematic diagram of the reverse propagation process in one embodiment;

[0034] Figure 5 A schematic diagram of a resource-constrained large model heterogeneous training method in another embodiment;

[0035] Figure 6 A comparison diagram of Zero-offload and a resource-constrained large model heterogeneous training method in an embodiment;

[0036] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] In one embodiment, Figure 1 As shown, a resource-constrained large model heterogeneous training method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0039] Step S102, among the activation quantities generated when the image processor performs forward propagation of the model, a first activation quantity is determined according to the dependency relationship between the calculation quantities corresponding to each of the activation quantities and / or the calculation quantity and storage quantity of each of the activation quantities, and the first activation quantity is asynchronously transmitted to the central processing unit.

[0040] The model is a machine learning model to be trained. Asynchronous transmission means that each time a first activation is generated, the first activation can be transmitted to the central processing unit without blocking the forward propagation.

[0041] Optionally, by analyzing the dependency between the computational amount in the forward propagation, back propagation, and model training computational graphs, the activation amount that needs to be recalculated based on the initial stage of the model or the shallower data of the model during subsequent model training if it is arbitrarily unloaded or discarded is obtained, and this type of activation amount that cannot be arbitrarily unloaded or discarded is used as the first activation amount. Alternatively, when the computational amount corresponding to the activation amount is greater than the corresponding specified value and the storage amount is greater than the corresponding specified value, the activation amount is used as the first activation amount. Alternatively, the union of the activation amount determined according to the dependency and the activation amount determined according to the computational amount and the storage amount is used as the first activation amount to reduce the video memory occupation of the activation amount during the forward propagation process, as well as the computational amount and video memory overhead of the activation amount during the back propagation process.

[0042] Optionally, the image processor trains the model based on preset training data, selects the first activation amount from the activation amounts generated during the forward propagation of the model, asynchronously transmits the first activation amount to the central processor, and discards the unselected activation amounts.

[0043] Step S104, loading the first activation amount from the central processing unit to the image processor during the model back propagation process.

[0044] In the process of model back propagation, the image processor needs to calculate the gradient of the model parameters based on the activation amount, so the activation amount is loaded. Optionally, according to the real-time computing requirements of the image processor during the back propagation process, the required first activation amount is determined, and the corresponding first activation amount is loaded from the central processor to the image processor in real time through the I / O port between the central processor and the image processor.

[0045] Step S106, obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount, and asynchronously transmitting the gradient to the central processing unit.

[0046] The gradient of the model parameters is the data calculated during the back propagation of the model. The adjustment direction of the model parameters can be determined by the gradient of the model parameters to achieve the update of the model parameters. Optionally, the image processor calculates the gradient of each layer of the model from back to front according to the chain derivation rule to obtain the gradient of the model parameters. After the image processor calculates the gradient of the model parameters, the gradient can be transmitted to the central processing unit in real time without waiting for the back propagation to be completed.

[0047] Step S108, determining the optimizer obtained based on the gradient update in the central processing unit, and obtaining the model parameters adjusted by the updated optimizer, and asynchronously transmitting the model parameters to the image processor.

[0048] The optimizer is used to adjust the weights and parameters of the model according to the gradient, so as to improve the performance of the model on the training data set. The optimizer can be an existing optimizer, such as SGD (Stochastic Gradient Descent), Adam (Adaptive Moment Estimation), etc., which are not limited here. Optionally, after the optimizer in the central processing unit updates the optimizer parameters and model weights based on the gradient, the updated model weights are transmitted to the image processor in real time.

[0049] In the above resource-constrained large model heterogeneous training method, the activations generated during the forward propagation of the model are selectively unloaded to the CPU (Central Processing Unit), and the required activations are loaded from the CPU during the backward propagation of the GPU (Graphics Processing Unit), so that the GPU calculates the parameter gradients based on the activations, reducing the extra time caused by the unloading, loading, and recalculation of the activations, and reducing the resources required by the GPU during large model training. By asynchronously transmitting data between the CPU and the GPU, the computational efficiency of large model training is improved, achieving the effect of balancing low graphics card resource consumption and high computational efficiency during large model training, and solving the problem of difficulty in training large models with limited video memory resources and low graphics card computational efficiency.

[0050] In one embodiment, among the activation quantities generated when the image processor performs forward propagation of the model, a first activation quantity is determined based on the dependency relationship between the calculation quantities corresponding to each activation quantity and / or the calculation quantity and storage quantity of each activation quantity, and the first activation quantity is asynchronously transmitted to the central processing unit, including: during the forward propagation of the model, the activation quantity corresponding to the input of the model, the activation quantity calculated depending on redundant items, and the activation quantity whose calculation quantity and storage quantity are respectively greater than the corresponding specified values ​​are taken as the first activation quantity.

[0051] Among them, if the activation amount calculated by relying on redundant items is discarded, the model needs to be recalculated to obtain redundant items during back propagation, which reduces the calculation efficiency. At the same time, the model relies on the input of the model for recalculation during back propagation. Using these two types of activation amounts and the activation amount whose calculation amount and storage amount are respectively greater than the corresponding specified values ​​as the first activation amount can reduce the amount of calculation during model training and achieve the effect of reducing video memory overhead.

[0052] Optionally, when the model includes multiple Transformer-blocks, and the Transformer-block includes a self-attention module and a multi-layer perception module, the first activation amount includes the input of the self-attention module during the forward propagation of the model, the activation amount that has a dependency relationship with the input of the self-attention module and the multi-layer perception module, and the triplet calculated based on the self-attention mechanism and the activation amount generated by the linear layer of the multi-layer perception module to obtain the first activation amount.

[0053] Exemplarily, the Transformer architecture is composed of multiple stacked Transformer-blocks, and the structures and parameter quantities of the multiple Transformer-blocks are the same. Figure 2 A schematic diagram of the activation and computational effort generated by the forward propagation of a Transformer-block is provided, as Figure 2 As shown, the Transformer-block includes a self-attention module and a multi-layer perception module MLP. Assume that the input matrix shape is a tensor of [batch_size B, sequence_length L, hidden_size H]. The triples calculated based on the self-attention mechanism include the corresponding Query vector, Key vector, and Value vector generated by three different linear transformations in the self-attention module. Among them, the activations generated by the linear layer of the Query vector, Key vector, Value vector, and the multi-layer perception module are activations whose calculation amount and storage amount are respectively greater than the corresponding specified values. The input of the self-attention module is the above-mentioned activation corresponding to the input of the model. The activation that has a dependency relationship with the input of the self-attention module and the multi-layer perception module is the above-mentioned activation calculated based on the redundant terms.

[0054] This setting is because in the forward propagation process, for each layer of Transformer-block, the calculation amount and activation amount generated in different calculation steps are different, mainly including the Attention of the self-attention module and the MLP calculation of the multi-layer perception module. The sequence length of Transformer training is recorded as L, and the size of the hidden layer is recorded as H. In general, H is much larger than L, so the amount of calculation of the Query vector, Key vector, and Value vector is relatively large and occupies more activations; the amount of calculation and activation of the MLP layer are both very large, and the amount of calculation of other operations such as Add, Layernorm, and Softmax is relatively small. At the same time, Figure 3 A schematic diagram of the dependencies in a Transformer-block is provided, such as Figure 3As shown in the figure, by analyzing the amount of computation in the forward propagation process, the amount of computation in the backward propagation process, and the dependencies in the computational graph, we can see that there are two Add operations and Layernorm operations in the Transformer-block. Among them, the Add operation depends not only on the output of the previous layer, but also on redundant items. If the activation generated by the Add operation of the forward propagation is discarded, then it is necessary to recalculate from the input X or the output generated by the first Layernorm, otherwise the backward propagation cannot be completed. Therefore, this operation will limit the corresponding activation and cannot be unloaded and discarded at will. Based on this, retain Figure 2 The activations generated by the two Add operations in the forward propagation, the input X, the Query vector, the Key vector, the Value vector, and the activations generated by the two linear layers in the MLP; the other activations are discarded and recalculated during backpropagation, thereby reducing the memory usage of the activations during forward propagation, as well as the computational effort and memory overhead of re-measuring based on the activations during backpropagation.

[0055] The following is an analysis of the activation overhead and computational complexity introduced by unloading the first activation to the CPU and discarding the remaining activations during forward propagation. The sequence length of Transformer training is denoted as L, the hidden layer size is denoted as H, and the batch size is denoted as B. Assume that the computing speed of the GPU is f (flops, floating point operations per second), and the bandwidth between the GPU and the CPU is bw (bandwidth). In addition, for ease of analysis, it is assumed that the communication between the CPU and the GPU is synchronized with the GPU's calculation, that is, only one of the operations can be performed at the same time until completion.

[0056] During the forward propagation of a Transformer-block, the activation overhead and computational complexity introduced by the calculation of each layer of the Transformer-block are shown in Table 1.

[0057] Table 1

[0058] layer X QUR <h2 style=";text-align:left;direction:ltr"><![CDATA[QK <h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> / sqrt(d)]]><h2 style=";text-align:left;direction:ltr"> Softmax (Softmax)V Activation quantity occupied BLH BLH <![CDATA[BL 2 ]]> <![CDATA[BL 2 ]]> BLH Calculation Amount 0 <![CDATA[BLH 2 ]]> <![CDATA[BL 2 H]]> <![CDATA[3BL 2 ]]> <![CDATA[BL 2 H]]> layer ADD Layernorm MLP-Linear1 Gelu MLP-Linear2 Activation quantity occupied BLH BLH 4BLH 4BLH BLH Calculation Amount BLH 4BLH <![CDATA[4BLH 2 ]]> 4BLH <![CDATA[4BLH 2 ]]>

[0059] Based on Table 1, we can calculate that the time required for a Transformer-block forward propagation is:

[0060]

[0061] Additional time overhead introduced by uninstalling activations:

[0062]

[0063] The maximum memory cost is the maximum activation amount generated by the calculation:

[0064] Mpeak =4BLH

[0065] In a Transformer-block, t introduced during the forward propagation offload Expenses as a percentage of total forward The ratio is:

[0066]

[0067] Generally, the number of hidden layers in large models ranges from 4096 to 12288, the sequence length ranges from 512 to 4196, and the size of H is 1-10 times larger than L. The t introduced in the forward propagation process of large models offload Expenses as a percentage of total forward The ratio is:

[0068]

[0069] The hidden layer size H, training sequence length L, and number of Transformer-blocks of the mainstream large models are shown in Table 2:

[0070] Table 2

[0071]

[0072] In addition, the computing speed of the GPU is 100 to 1000 times the communication speed between the GPU and the CPU. In general, the additional time overhead introduced is between 10% and 50%. Therefore, based on the solution in this embodiment, selective activation unloading and recalculation can enable the model to trade a large amount of video memory overhead for a small additional time overhead during the forward propagation process.

[0073] In one embodiment, the model includes multiple basic building units, and the image processor obtains the gradient of the model parameters calculated based on the first activation amount, and asynchronously transmits the gradient to the central processing unit, including: whenever the image processor calculates the gradient of the parameter of the basic building unit based on the first activation amount, asynchronously transmitting the gradient corresponding to the current basic building unit to the central processing unit in units of basic building units.

[0074] The model is composed of multiple basic building blocks stacked together. The basic structural unit can be constructed based on a single network layer in the model, or a sub-network or module composed of multiple network layers in the model can be used as the basic building unit. For example, when the model is a Transformer model, the multiple basic building units included in the model are Transformer-blocks, including self-attention modules and multi-layer perception modules.

[0075] Optionally, when the model is back-propagated, the gradient calculation is performed from back to front according to the connection order of multiple basic building blocks in the model. Whenever the parameter gradient calculation of a basic building block is completed, the parameter gradient corresponding to the basic building block is transmitted to the central processing unit. In this way, the central processing unit can receive the parameter gradient in real time, and the central processing unit can update the optimizer parameters corresponding to the current basic building block in real time according to the received gradient, thereby improving the model training efficiency.

[0076] Furthermore, after obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount, the method also includes: releasing the second activation amount used by the image processor when calculating the gradient of the model parameter based on the first activation amount. The second activation amount is the activation amount used when the image processor calculates the gradient of the model parameter. Optionally, each time the gradient of a parameter matrix in the basic building unit is calculated, the activation amount involved in calculating the gradient can be used as the second activation amount, and the second activation amount can be released. In this embodiment, by releasing the second activation amount when calculating the gradient, the video memory resources required for model training can be reduced.

[0077] For ease of understanding, the following is an analysis of the amount of computation required to recalculate the activations introduced by backpropagation, the loading overhead of the activations, and the memory overhead of the activations. Figure 2 In the Transformer model shown, different layers require different amounts of computation to calculate gradients during back-propagation, and the back-propagation process of the Transformer-block is slightly more complicated than the forward propagation. When deriving the parameters or activations of a certain layer, unlike forward propagation, which only requires the activations of the previous layer, back-propagation requires the gradients passed down from the next layer, the activations of the previous layer, and its own parameters. In some cases, it also requires its own activations such as Relu, Sigmod, and tanh. The computation time of the back-propagation process is t backward Is a fixed value:

[0078]

[0079] Activation loading time t reload , activation amount recalculation time t recompute , and the activation amount and peak memory usage M peak Conduct analysis.

[0080] Figure 4 A schematic diagram of the back propagation process is provided, such as Figure 4 As shown, the following steps are included:

[0081] Step S401, load the second Add, recalculate the second Layernorm and output, and then back propagate to calculate the gradient δ(A) of Add ADD2).

[0082] Among them, the second Add refers to the Add corresponding to the MLP output, and the second Layernorm refers to the Layernorm based on the second Add output. Figure 2 The O connected to the second Layernorm is the input X of the next Transformer-block and does not need to be recalculated.

[0083] Activation amount loading time in step S401:

[0084]

[0085] Recalculation time:

[0086]

[0087] Gradient calculation process:

[0088]

[0089] The peak memory usage of the activation amount is the sum of all current activation amounts:

[0090] M peak =BLH×3=3BLH

[0091] Step S402, load the activations of the two linear layers of MLP and recalculate the Gelu layer, and then calculate the gradients of MLP-Linear2 and Gelu.

[0092] Activation amount loading time in step S402:

[0093]

[0094] Recalculation time:

[0095]

[0096] Gradient calculation process:

[0097]

[0098] Among them, A represents the gradient of the calculated activation amount, W represents the gradient of the calculated parameters, and the final calculated value is the calculated amount.

[0099] Activation amount video memory peak usage:

[0100] M peak =BLH+4BLH+4BLH=9BLH

[0101] Step S403, load the result of the first Add, recalculate the first Layernorm, and then calculate the gradient of MLP-Linear1 and the gradients of the first Layernorm and the first Add.

[0102] Among them, the first Add refers to the value obtained based on the Add operation corresponding to the self-attention output, and the first Layernorm refers to the value obtained based on the first Add.

[0103] Activation amount loading time in step S403:

[0104]

[0105] Recalculation time:

[0106]

[0107] The gradient calculation process is as follows:

[0108]

[0109] Activation amount video memory peak usage:

[0110] M peak =4BLH+BLH+BLH=6BLH

[0111] Step S404, load input X, vector KQV, recalculate Self-Attention, calculate QK T / sqrt(d), softMax and SV, calculate the gradients of all remaining activations and parameters.

[0112] Activation amount loading time in step S404:

[0113]

[0114] Recalculation time:

[0115]

[0116] Gradient calculation process:

[0117] Compute the gradient of the intermediate activation:

[0118]

[0119] Calculate W K , W Q , W V Gradient of the parameters:

[0120]

[0121]

[0122] Compute the gradient of input X:

[0123]

[0124] Activation amount video memory peak usage:

[0125] M peak =5BLH+2BL 2

[0126] The following is an overall analysis of the Transformer-block back-propagation process.

[0127] Among them, the overall loading time:

[0128]

[0129] Total weight calculation time:

[0130]

[0131] Peak memory usage:

[0132] M peak =max(5BLH+2BL 2 ,6BLH,9BLH,3BLH)=9BLH(H>L)

[0133] The additional time overhead introduced has the same complexity as the forward propagation:

[0134]

[0135] Reduced video memory usage overhead ratio:

[0136]

[0137] Therefore, when unloading, loading, and recalculating activations at the Transformer-block granularity, only one Transformer-block is performing backpropagation at a time during backpropagation, so the video memory overhead will be lower than the backpropagation overhead of traditional technologies. In addition, the above analysis assumes that the GPU calculation and the communication between the GPU and the CPU do not overlap. In actual situations, the GPU calculation and communication can overlap, so the additional time overhead is smaller.

[0138] In one embodiment, the central processing unit stores model parameters and an optimizer of a first precision, and the image processor stores model parameters of a second precision and the gradient of the model parameters under the second precision, and the first precision is higher than the second precision. Optionally, a 32-bit floating point number is used as the first precision, and a 16-bit floating point number is used as the second precision. The first precision and the second precision can also be set to other precision values. The optimizer is placed in the central processing unit, so that the update of the optimizer parameters is completed by the central processing unit, reducing the processing burden of the image processor. By reducing the precision of the model parameters and gradients in the image processor, resource consumption can be reduced.

[0139] Furthermore, the model parameters are asynchronously transmitted to the image processor, including: copying the model parameters based on the second precision; asynchronously transmitting the copied model parameters of the second precision to the image processor. Optionally, the precision of the model parameters is adjusted by a type conversion function, and the converted model parameters of the second precision are transmitted to the image processor in real time.

[0140] The precision of the model parameters stored in the image processor is associated with the consumption of video memory resources. In this embodiment, using a lower precision data type to store the model parameters in the image processor can reduce video memory consumption and speed up the operation.

[0141] In one embodiment, after transmitting the optimizer asynchronous parameters to the image processor, the method further includes: determining whether the parameters of the model have been updated; if not, repeating the following steps: asynchronously transmitting the first activation amount to the central processor during the model forward propagation process; loading the first activation amount from the central processor to the image processor during the model reverse propagation process; loading the first activation amount from the central processor to the image processor during the model reverse propagation process; obtaining the gradient of the model parameters calculated by the image processor based on the first activation amount, and asynchronously transmitting the gradient to the central processor; determining the optimizer obtained based on the gradient update in the central processor, and obtaining the model parameters adjusted by the updated optimizer, and asynchronously transmitting the model parameters to the image processor. Optionally, each time the steps of determining the optimizer, obtaining and transmitting the adjusted model parameters are performed, the method includes: obtaining the optimizer parameters of the central processor based on the gradient update model, obtaining the model parameters adjusted by the updated optimizer, and copying the adjusted model parameters with the second precision and asynchronously transmitting them to the image processor.

[0142] Among them, if the parameters of the model are updated, the process of executing forward propagation and backward propagation is stopped. Optionally, the transmission of the first activation amount, gradient and optimizer parameters is realized through the I / O between the GPU and the CPU. Optionally, when the model includes multiple basic building blocks, the gradient value is transmitted from the image processor to the central processing unit in units of basic building blocks during the backward propagation of the model, and the optimizer parameters are transmitted from the central processing unit to the image processor. By judging whether the parameters of each basic building block unit in the model are all updated, it is judged whether the parameters of the model are updated.

[0143] Optionally, Figure 5 A schematic diagram of another resource-constrained large model heterogeneous training method is provided, such as Figure 5 As shown, the following steps are included:

[0144] Step S501, model preparation, includes the following sub-steps:

[0145] Step S5011, dividing the model training process into three stages: forward propagation, back propagation and parameter update;

[0146] Step S5012, putting the model parameters and optimizer of the first precision into the CPU memory, and putting the model parameters of the second precision and the half-precision gradients of the parameters into the GPU.

[0147] Step S502, the forward propagation process, includes the following sub-steps:

[0148] Step S5021, during the forward propagation process, the input will pass through multiple basic building blocks to obtain the output, and only the first activation amount in the forward propagation process is retained.

[0149] Optionally, the model includes multiple Transformer-blocks, and the input passes through multiple Transformer-blocks and then passes through an Embedding layer to obtain the output. For each Transformer-block, the activations generated by the two Add operations and the input X, as well as the activations generated by K, Q, V, and the two linear layers in the MLP are taken as the first activations, and the first activations are retained;

[0150] Step S5022: After the first activation amount to be saved is calculated, the first activation amount is asynchronously transmitted to the CPU.

[0151] Step S503, the back propagation process, includes the following sub-steps:

[0152] Step S5031, when it is necessary to update the parameters in a certain basic building unit, asynchronously load the unloaded first activation amount into the GPU;

[0153] Step S5032, recalculating based on the uninstalled first activation amount;

[0154] Step S5033, calculate the gradient of the parameters in the current basic building unit, and immediately discard the second activation involved in the calculation process after calculating the gradient of a parameter matrix.

[0155] Step S504, asynchronous parameter update and transfer, includes the following sub-steps:

[0156] Step S5041, during the back propagation process, whenever the gradient calculation of a basic building block is completed, the gradient is asynchronously transmitted to the CPU;

[0157] Step S5042, after the gradient transmission of each basic building block is completed, the CPU completes the update of the optimizer and parameters corresponding to the Transformer-block;

[0158] Step S5043, whenever the parameter update of a basic building block is completed, the half-precision parameters of the basic building block are asynchronously transmitted to the GPU;

[0159] Step S5044: The GPU updates the parameters of the current basic building unit until all basic building units are updated. When the parameters of the current basic building unit are updated, the storage location of the basic building unit on the GPU remains unchanged.

[0160] It should be understood that, although the steps in the flowcharts involved in the embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages, which are not necessarily completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps. For example, while step S5033 performs the gradient calculation of the parameter matrix, step S5041 can be executed based on the calculated gradient to asynchronously transmit the gradient to the CPU; while step S5031 asynchronously loads the unloaded first activation amount into the GPU, step S5032 can be synchronously recalculated based on the acquired first activation amount.

[0161] Furthermore, in one embodiment, the method includes: dividing the training data of the model to obtain multiple training sub-data; traversing the multiple training sub-data, inputting the traversed training sub-data into the model, and performing forward propagation and backward propagation on the model based on the training sub-data set until the traversal of the multiple training sub-data is completed.

[0162] Optionally, after the traversed training sub-data is input into the model and all the parameters of the model are updated, the next training sub-data is traversed. The newly traversed training sub-data is input into the model after the parameters are updated. When all the training sub-data have completed the training, it is determined that one round (epoch) of model training is completed.

[0163] It should be understood that the parameter update and transfer in this embodiment is different from the related technology Zero-offload. Zero-offload asynchronously unloads the gradient to the CPU during the back-propagation process. The CPU does not update the parameters immediately, but starts to update the parameters after the gradient is completely unloaded, and the computing resources of the GPU are almost not used in this process. Figure 6 Comparison chart of Zero-offload and the resource-constrained large model heterogeneous training method in this embodiment. Figure 6 It can be seen that the resource-constrained large model heterogeneous training method in this embodiment uses asynchronous parameter update and transmission in units of Transformer-block: during the back-propagation process, the calculated parameter gradients are asynchronously transmitted to the CPU in units of one Transformer-block, and the CPU completes the update of the optimizer and parameters. Whenever the parameter update of a Transformer-block is completed, the half-precision parameters are immediately asynchronously transmitted to the GPU to complete the update of the half-precision parameters in the GPU, preparing for the next step of training. Compared with the traditional method, the model training method in this embodiment saves more video memory overhead and computing overhead, overlaps the calculation, gradient unloading, parameter update in the CPU, and parameter update in the GPU in the back-propagation process, makes more full use of computing resources and communication resources, and improves computing efficiency.

[0164] In one embodiment, reference Figure 2The Transformer model shown proposes another large model heterogeneous training method suitable for resource-constrained conditions. This embodiment uses GPU and CPU to perform large model heterogeneous training optimization based on the Transformer architecture. After obtaining the preset training data, prepare the model, including: splitting the training data into n batches to obtain n training sub-data. Put the full-precision model parameters and optimizer into the CPU memory, and put the half-precision model parameters and the half-precision gradients of the parameters into the GPU. After completing the model preparation, perform the steps of forward propagation, backpropagation, gradient calculation, and parameter update for each batch in each round of training.

[0165] During forward propagation, for each Transformer-block, the KQV matrix is ​​first calculated, where K = XW k , Q = XW q , V = XW v After the calculation, the X and KQV matrices are asynchronously transmitted to the CPU. Next, Attention is calculated:

[0166]

[0167] After getting Attention, enter the Add layer and Layernorm layer:

[0168] add1=Attention(X)+X

[0169] ln1=Layernorm(add1)

[0170] The output of add1 from the Add layer is asynchronously unloaded to the CPU. Next, the output parameters are calculated based on the MLP module:

[0171] o3=MLP-Linear1(output2)

[0172] o4=Gelu(o3)

[0173] o5=MLP-Linear2(o4)

[0174] After o3 and o5 are asynchronously unloaded to the CPU after calculation, they go through an Add layer and a Layernorm layer again:

[0175] add2=MLP(ln1)+ln1

[0176] ln2=Layer norm(add2)

[0177] Unload add2 asynchronously to the CPU, and the forward propagation of a Transformer-block is completed.

[0178] During back propagation and gradient calculation, for each Transformer-block, the gradient calculation is performed from back to front according to the chain rule. Taking the update of the parameters of the nth Transformer-block as an example, the update method includes: loading the activation of add2 and recalculating the second Layernorm and output; back propagating to calculate the gradient of add2; when calculating the gradient, the activation of the two linear layers of MLP can be loaded and the Gelu layer can be recalculated. When the second Add layer is calculated and the activation is recalculated, the gradient of MLP-Linear2 and Gelu can be calculated; loading the activation of the first Add in the nth Transformer-block and recalculating the first Layernorm, calculating the gradient of MLP-Linear1 and the gradient of the first Layernorm and the first Add; loading the input X and KQV and recalculating Self-Attention, calculating QK / sqrt(d), softMax and SV, and calculating the gradient of all remaining activations and parameters.

[0179] Optionally, while computing the remaining parameter gradients, the activations of the n-1th Transformer-block can be asynchronously loaded and recalculated based on the loaded activations.

[0180] Parameter update includes: when performing the above back propagation, after loading the activation of the first Add in the nth Transformer-block and recalculating the first Layernorm, calculating the gradient of MLP-Linear1 and the gradient of the first Layernorm and the first Add, the GPU can immediately unload the gradient of the block parameters to the CPU. The CPU updates the optimizer state and model parameters of the block based on the parameter gradient; after the model parameters are updated, the model parameters are semi-precisioned and transmitted to the GPU. The GPU updates the parameters. Repeat the above steps until all parameters are updated.

[0181] Input a new batch to the model to perform forward propagation, back propagation, gradient calculation, and parameter update steps again to complete a new round of training. This process ends when all batches have completed training, completing one epoch of training.

[0182] In the related technology, strategies that can train and deploy large models under resource-constrained conditions include: activation recalculation, activation unloading, and CPU and GPU heterogeneous training. Among them, activation recalculation only retains part of the activation during the calculation process, and the other activations are recalculated through checkpoints when they are needed. This method will cause most of the time in the back propagation process to be spent on recalculating activation values, reducing the speed of back propagation. Activation unloading unloads the activation from the GPU to the host memory during the forward propagation calculation process, and then reloads it to the GPU to calculate the gradient during the back propagation process. This method only utilizes the host's storage resources and does not utilize the CPU's computing resources.

[0183] Using CPU and GPU heterogeneous training, such as Zero-Offload, stores the optimizer and full-precision parameters in the CPU, and copies a half-precision parameter to the GPU for calculation. Finally, the gradient is passed to the CPU to update the optimizer and full-precision parameters. This method is currently the most widely used method. However, most of the time on a single GPU is spent on communication between the GPU and the CPU, resulting in too low GPU computing efficiency. The subsequent PatrickStar manages and stores model data in blocks in a heterogeneous space, and Mobius splits the model into multiple segments for segmented loading and unloading, achieving efficient large model training on a few commercial GPU clusters. However, these methods are still mainly aimed at distributed clusters and on single machines with more limited resources.

[0184] For selective activation unloading, retention, and recalculation, Pofo proposed a dynamic programming method to find the best strategy for activation unloading and recalculation. This method targets the traditional neural network training process. For each layer of the network, it chooses one of the three methods: unloading activations, retaining activations, or discarding activations. Finally, an execution sequence is formed to make the peak memory usage as low as possible and the overall training time as short as possible. However, this method is too theoretical and difficult to apply to actual complex computing scenarios. It is mainly suitable for serialized networks with no special dependencies between traditional layers. The Transformer's computational graph is more complex and there are some cross-layer dependencies between different model layers. Figure 2For example, Self-Attention and its subsequent Add layer need to rely on input X, and the second Add layer needs to rely on the output of the first Layernorm, which makes the pofo algorithm unable to adapt well to the Transformer architecture. In contrast, a selective activation recalculation method suitable for distributed training environments is proposed for the Transformer architecture. This method first estimates the size of the activation generated in the forward propagation. In the parallel multi-machine sequence, it is found that as the sequence grows, the activation generated by Attention is large but the computation is small. Therefore, the activation generated by the Attention process is discarded, and other activations are retained in the device and recalculated during back propagation. However, this method is mainly suitable for distributed training scenarios with extremely long sequences, and does not consider the activation unloading strategy. It is not suitable for single-machine training environments with limited GPU resources.

[0185] The method of using heterogeneous storage to train a large model under limited resources in this embodiment is based on a heterogeneous system composed of a single GPU and a CPU. In the training process of the large model, the activation amount, gradient calculation, unloading and loading, and activation amount checkpoint are realized. Among them, the activation amount checkpoint includes activation amount discarding and recalculation. In general, the method in this embodiment can exchange a large amount of video memory overhead for a small additional time overhead during the forward propagation and reverse propagation of the model. Among them, the additional time overhead caused by unloading, loading, and recalculation of activation amount is between 10%-50%, and more than 50% of video memory overhead can be saved in a single Transformer-block. Since the analysis process is an analysis of a Transformer-block, a large model is generally composed of a sequence of 20 to 100 Transformer-blocks. During reverse propagation, only one Transformer-block is performing reverse propagation at a time, so the video memory overhead will be more than 95% lower than the original approach. Therefore, the method in this embodiment can complete the training of a large model with a video memory requirement greater than the physical video memory when the video memory resources are very limited, and ensure the computing efficiency of the graphics card.

[0186] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless method can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a large model heterogeneous training method with limited resources is implemented.

[0187] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0189] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0190] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A resource-constrained large model heterogeneous training method, characterized in that: The method comprises: Among the activations generated when the image processor performs forward propagation of the model, a first activation is determined according to a dependency relationship between calculation amounts corresponding to each of the activations and / or a calculation amount and storage amount of each of the activations, and the first activation is asynchronously transmitted to the central processing unit; loading the first activation from the central processing unit to the image processor during the back propagation of the model; Obtaining a gradient of a model parameter calculated by the image processor based on the first activation amount, and asynchronously transmitting the gradient to the central processing unit; An optimizer updated based on the gradient in the central processing unit is determined, and model parameters adjusted by the updated optimizer are obtained, and the model parameters are asynchronously transmitted to the image processor.

2. The method according to claim 1, characterized in that The model includes a plurality of basic building blocks, and the step of obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount and asynchronously transmitting the gradient to the central processing unit includes: Whenever the image processor calculates the gradient of the parameter of the basic building unit based on the first activation amount, the gradient corresponding to the current basic building unit is asynchronously transmitted to the central processing unit in units of the basic building unit.

3. The method according to claim 1 or 2, characterized in that: After obtaining the gradient of the model parameter calculated by the image processor based on the first activation amount, the method further includes: A second activation amount used by the image processor when calculating a gradient of a model parameter based on the first activation amount is released.

4. The method according to claim 1, characterized in that: The method comprises: determining a first activation amount among the activation amounts generated when the image processor performs forward propagation of the model according to the dependency relationship between the calculation amounts corresponding to the activation amounts and / or the calculation amount and storage amount of each activation amount, and asynchronously transmitting the first activation amount to the central processing unit, including: In the process of forward propagation of the model, the activation amount corresponding to the input of the model, the activation amount calculated based on the redundant term, and the activation amount whose calculation amount and storage amount are respectively greater than the corresponding specified value are used as the first activation amount; The first activation amount is asynchronously transmitted to the central processing unit.

5. The method according to claim 1, characterized in that The central processor stores model parameters and an optimizer of a first precision, and the image processor stores model parameters of a second precision and gradients of the model parameters under the second precision, wherein the first precision is higher than the second precision.

6. The method according to claim 5, characterized in that The step of asynchronously transmitting the model parameters to the image processor comprises: replicating the model parameters based on the second precision; The model parameters of the second precision obtained by asynchronous transmission are copied to the image processor.

7. The method according to claim 1, characterized in that After transmitting the optimizer asynchronous parameters to the image processor, the method further includes: Determine whether the parameters of the model have been updated; If not, repeat the following steps: asynchronously transmit the first activation amount to the central processing unit during the forward propagation of the model; load the first activation amount from the central processing unit to the image processor during the reverse propagation of the model; load the first activation amount from the central processing unit to the image processor during the reverse propagation of the model; obtain the gradient of the model parameters calculated by the image processor based on the first activation amount, and asynchronously transmit the gradient to the central processing unit; determine the optimizer updated based on the gradient in the central processing unit, and obtain the model parameters adjusted by the updated optimizer, and asynchronously transmit the model parameters to the image processor.

8. The method according to claim 7, characterized in that include: Dividing the training data of the model to obtain multiple training sub-data; The plurality of training sub-data are traversed, the traversed training sub-data are input into the model, and forward propagation and backward propagation are performed on the model based on the training sub-data set until the traversal of the plurality of training sub-data is completed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images