Model training task execution methods, graphics processors, and model training systems

Abstract

This application relates to a model training task execution method, a graphics processing unit (GPU), and a model training system. The model training task execution method includes: when performing attention calculations for the forward propagation of the model on the current text segment, sequentially loading the key-value pair activation values ​​corresponding to each preceding text segment from the central processing unit (CPU) for attention calculation; simultaneously unloading the key-value pair activation values ​​corresponding to the current text segment to the CPU; when performing attention calculations for the backpropagation of the model on the current text segment, loading the forward propagation output and key-value pair activation values ​​corresponding to the current text segment from the CPU, and sequentially loading the attention activation values ​​corresponding to each preceding text segment, performing attention calculations with the forward propagation output and key-value pair activation values ​​corresponding to the current text segment to obtain gradient values; and unloading the gradient values ​​to the CPU. This method can improve the execution efficiency of text training tasks for large language models under GPU resource constraints.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to methods for performing model training tasks, graphics processors, and model training systems. Background Technology

[0002] To improve the ability of large language models, such as Generative Pretrained Transformation Model (GPT), AI Large Language Model (Llama), and General Language Model (GLM), to understand and process long texts, such as entire books or contextual dialogues with more than 32K tokens, it is necessary to input such long texts (hereinafter referred to as long sequences) into the large language model for training.

[0003] The training process includes iterative text input, model computation, and model parameter updates. Model computation is divided into two processes: forward propagation and backward propagation. During forward propagation, each module of the large language model sequentially calculates its output based on the input text content; these outputs are called activation values ​​and need to be retained. During backward propagation, these activation values ​​are used for further computation, updating the model parameters of the large language model. Thus, it is known that the activation values ​​generated during forward propagation need to be stored in the GPU's video memory, and the space occupied by these activation values ​​increases with the length of the text. When the length of the text input to the large language model reaches tens of thousands or even millions of tokens, the GPU memory required for the generated activation values ​​will exceed the memory capacity of a single GPU.

[0004] In some related technologies, distributed sequence parallel training is often employed, where the activation values ​​of long sequences are divided into multiple shorter sequences and placed on multiple GPUs on a single server or multiple GPUs across multiple servers in a cluster. These GPUs exchange data and perform parallel computations. This approach requires a large number of GPUs to maintain training, and communication between multiple GPUs and even multiple servers is necessary. Therefore, it places significant demands on hardware and communication, making it unsuitable for efficient training of large language models under GPU resource constraints. Other related technologies utilize activation value recalculation techniques, increasing computational requirements to reduce GPU memory requirements for model training. However, the overall efficiency of these methods is reduced by the additional computational demands of activation value recalculation. Furthermore, recalculation techniques still require retaining complete activation values ​​for some model modules, leading to increased GPU memory usage as sequence length increases. Therefore, current related technologies exhibit low efficiency in training large language models for text under GPU resource constraints.

[0005] Therefore, no effective solution has yet been proposed to address the problem of low execution efficiency for text training tasks of large language models under GPU resource constraints in related technologies. Summary of the Invention

[0006] This embodiment provides a model training task execution method, a graphics processor, and a model training system to solve the problem of low execution efficiency of text training tasks for large language models under GPU resource constraints in related technologies.

[0007] Firstly, this embodiment provides a method for executing a model training task, for a graphics processor, the method comprising:

[0008] Obtain the current text segment; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in order;

[0009] When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit.

[0010] When performing backpropagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the forward propagation output and key-value pair activation values ​​corresponding to the current text segment are loaded from the central processing unit, and the attention activation values ​​corresponding to each preceding text segment are loaded sequentially. Attention calculation is then performed with the forward propagation output and key-value pair activation values ​​corresponding to the current text segment to obtain gradient values; the gradient values ​​are then unloaded into the central processing unit.

[0011] In some embodiments, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation, including:

[0012] If the current text segment has at least two preceding text segments, then all preceding text segments other than the first text segment are taken as the target preceding text segments in sequence;

[0013] While performing attention calculation based on the activation value of the key-value pair corresponding to the target preceding text segment, the activation value of the key-value pair corresponding to the previous text segment of the target preceding text segment is loaded;

[0014] After performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, delete the key-value pair activation value corresponding to the target preceding text segment.

[0015] In some embodiments, when performing model forward propagation on the current text segment, the method further includes:

[0016] If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0017] In some embodiments, when performing model backpropagation on the current text segment, the method further includes:

[0018] If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment of the current text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0019] In some embodiments, after all text segments of the target text sequence have completed backpropagation, the method further includes:

[0020] Based on the backpropagation results of all text segments of the target text sequence, the model parameters of the trained model and the optimizer state of the model are updated.

[0021] Secondly, this embodiment provides a model training task execution method for a central processing unit, the method comprising:

[0022] In response to the graphics processor performing attention calculations in the forward propagation of the model on the current text segment, the system sequentially returns each first read value to the graphics processor for each first read value in a first loading operation; and receives and stores the key-value pair activation values ​​corresponding to the current text segment unloaded by the graphics processor; wherein each first read value is a key-value pair activation value corresponding to each preceding text segment of the current text segment; the first loading operation is initiated by the graphics processor when it determines that the current text segment has a preceding text segment; the current text segment is one of the text segments after the target text sequence is segmented sequentially;

[0023] In response to the graphics processor performing attention calculations for backpropagation of the model on the current text segment, the second read value is returned to the graphics processor for a second loading operation on the second read value; and in response to the graphics processor performing third loading operations on each of the first read values ​​in sequence, each of the first read values ​​is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and key-value pair activation value corresponding to the current text segment; the gradient value unloaded by the graphics processor is received and stored; the gradient value is obtained by performing attention calculations based on the second read value and each of the first read values ​​respectively.

[0024] In some embodiments, the method further includes, prior to the graphics processor performing the forward propagation of the model:

[0025] The maximum segment size that the graphics processor can accommodate is determined based on the model initialization parameters corresponding to the model to be trained;

[0026] Based on the maximum segment size, the target text sequence is segmented to obtain each text segment.

[0027] In some embodiments, the method further includes:

[0028] The required memory capacity of the central processing unit is determined based on the model initialization parameters, the length of the target text sequence, and the maximum segment size.

[0029] When the required memory capacity is greater than the available memory capacity of the central processing unit, a recalculation strategy is set for the model to be trained.

[0030] Thirdly, this embodiment provides a graphics processor, including: an acquisition module, a forward propagation module, and a backward propagation module;

[0031] The acquisition module is used to acquire the current text segment; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in sequence;

[0032] The forward propagation module is used to, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, load the key-value pair activation values ​​corresponding to each preceding text segment sequentially from the central processing unit for attention calculation during attention calculation; at the same time, unload the key-value pair activation values ​​corresponding to the current text segment to the central processing unit.

[0033] The backpropagation module is configured to, when performing model backpropagation on the current text segment, if the current text segment has a preceding text segment, load the forward propagation output result and key-value pair activation value corresponding to the current text segment from the central processing unit during attention calculation, and sequentially load the attention activation value corresponding to each preceding text segment, perform attention calculation with the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain a gradient value; and unload the gradient value to the central processing unit.

[0034] Fourthly, this embodiment provides a model training system, including: a graphics processor and a central processing unit;

[0035] The graphics processor is used to execute the model training task execution method described in the first aspect above;

[0036] The central processing unit is used to execute the model training task execution method described in the second aspect above.

[0037] Compared with related technologies, this embodiment provides a model training task execution method, a graphics processor, and a model training system. The model training task execution method involves obtaining the current text segment, which is one of the text segments after the target text sequence is segmented sequentially. During forward propagation of the model on the current text segment, if a preceding text segment exists, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit (CPU) for attention calculation. Simultaneously, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the CPU. During backward propagation of the model on the current text segment, if a preceding text segment exists, the forward propagation output and key-value pair activation values ​​corresponding to the current text segment are loaded from the CPU for attention calculation. The attention activation values ​​corresponding to each preceding text segment are then sequentially loaded and combined with the forward propagation output and key-value pair activation values ​​corresponding to the current text segment for attention calculation to obtain gradient values. The gradient values ​​are then unloaded to the CPU. This method overcomes the GPU memory bottleneck in text sequence training and overlaps the transmission time overhead with the computation time overhead, thereby improving the execution efficiency of large language model text training tasks under GPU resource constraints.

[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of the terminal for the model training task execution method according to an embodiment of this application;

[0041] Figure 2 This is a flowchart of the model training task execution method according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram illustrating the dependency relationships between various text segments in the model calculation of this application embodiment;

[0043] Figure 4 This is a flowchart of the model training task execution method according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the structural segmentation of a large language model according to an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of one type of activation value transmission according to an embodiment of this application;

[0046] Figure 7 This is a partial structural diagram of a large language model according to an embodiment of this application;

[0047] Figure 8 This is a schematic diagram illustrating a combination of segmented calculation and activation value transmission in an embodiment of this application;

[0048] Figure 9 This is a flowchart of a model training task execution method according to some embodiments of this application;

[0049] Figure 10 This is a structural block diagram of the graphics processor according to an embodiment of this application;

[0050] Figure 11 This is a schematic diagram of the structure of the model training system according to an embodiment of this application. Detailed Implementation

[0051] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0053] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the model training task execution method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0054] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model training task execution method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0056] This embodiment provides a model training task execution method for a graphics processing unit. Figure 2 This is a flowchart of the model training task execution method in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0057] Step S210: Obtain the current text segment; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in order.

[0058] The target text sequence can be a pre-determined text sequence needed for training a large language model. For example, the target text sequence can be a text sequence with a length exceeding 128K tokens. Before training, the target text sequence can be sequentially divided into several text segments. Specifically, the length of each segment can be made the same, and the length of each segment can be a power of 2 (to facilitate GPU computation and transmission). If the length of the target text sequence itself cannot satisfy the requirement that the length of each segment is a power of 2, a special token can be added to the end of the target text sequence to achieve the required length. For example, the target text sequence can be obtained from a dataset and sequentially divided into multiple text segments: Chunk1 to Chunk... nThis improves model training efficiency. Furthermore, it is understood that the model training task execution method in this embodiment is applicable to large language models.

[0059] Step S220: When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit.

[0060] Forward and backward propagation are the core processes in training a large language model, responsible for calculating prediction results and optimizing model parameters, respectively. The role of forward propagation is to process the input data layer by layer through the neural network to obtain the final predicted output. The role of backward propagation is to adjust the model parameters based on the prediction error, minimizing the loss function. Activation values ​​are intermediate results of the output of each layer in forward propagation. The gradient of an activation value is the partial derivative of the loss function with respect to the activation values ​​of a particular layer in backward propagation.

[0061] After obtaining the above segmentation, each text segment Chunk i After (i=1,2,...,n), when entering the forward propagation phase, it is necessary to process each text segment Chunk i (i=1,2,...,n), the forward propagation processes are executed sequentially from front to back (i from 1 to n). Specifically, for the current text segment Chunk... i The non-attention computations of each module in the large language model are executed sequentially until the attention computation module is encountered. Before the attention computation, word embedding, dropout, layer normalization, and query-key-value (QKV) projection are performed sequentially on the current text segment to obtain the query vector (Q... i ), key vector (K) i ) and value vector (V) i The key vector and value vector together form the key-value pair activation value (hereinafter referred to as KV activation value). In addition, other KV activation values ​​will also be generated. i V i Other activation values ​​besides these will be marked as... .

[0062] The preceding text segment of the current text segment can be any text segment that precedes the current text segment when the target text sequence is segmented sequentially. For example, if the current text segment is a Chunk... i Then the preceding text segment of the current text segment can include Chunk i-1To Chunk1. Apart from the first text segment Chunk1, the other text segments (Chunk2 to Chunk1) n Each of them has at least one preceding text segment.

[0063] Large language models process the entire target text sequence, thus requiring global sequence information for computation during training. To address this, the computation order can be set based on the dependencies between different text segments in the model's calculations, ensuring that the sequential processing of each segment does not affect the model's training performance. Figure 3 This diagram illustrates the dependencies between different text segments during model computation. For example... Figure 3 As shown, the calculation of the arrow's endpoint can only be performed after the calculation of the arrow's starting point is completed. The training process can be broadly divided into two stages: forward propagation and backward propagation. The calculations involved in these two stages can be based on... Figure 3 The arrows indicate that calculations are performed sequentially. These calculations can include attention-based and non-attention-based calculations. In the forward propagation, subsequent text segments require the KV activation values ​​calculated from the preceding text segments; therefore, the calculations proceed sequentially from segment 1 (the first text segment) to the last text segment (…). Figure 3 In section 3), forward propagation is performed sequentially. During backward propagation, since the KV activation value gradients calculated by the backward propagation of the preceding text segment are needed, backward propagation is performed sequentially from the last text segment to the first text segment.

[0064] When performing attention calculations on the current text segment, if the current text segment has a preceding text segment, the attention can be calculated based on the Q corresponding to the current text segment. i And K corresponding to the current text segment and each preceding text segment of the current text segment. j and V j Where j = i, i-1, ..., 1, the preset forward propagation function (e.g., the forward propagation function based on FlashAttention) is called sequentially to perform attention calculation. While performing attention calculation on the current text segment, the K corresponding to the current text segment can also be unloaded. i V i The result is then sent to the Central Processing Unit (CPU). Alternatively, algorithms such as online softmax can be used to update the result after each attention calculation in the current text segment, ultimately yielding the attention calculation output O. i .

[0065] Computation on the GPU primarily utilizes the GPU's computational units, not the bandwidth used for data transfer between the GPU and CPU. Offloading, however, mainly consumes the bandwidth between the GPU and CPU. Therefore, by simultaneously unloading key-value activations and performing attention calculations, computational units can perform calculations while data is being transferred, making the total time (data transfer time + computation time) closer to the computation time, thus masking the time overhead of data transfer. In this embodiment, unloading activations from the GPU refers to transferring them from the GPU's video memory to the CPU's main memory. Loading activations from the CPU refers to copying the corresponding activations from the CPU's memory to the GPU. After completing an attention calculation based on the loaded key-value activations in the GPU, these activations can be directly deleted from the GPU.

[0066] Understandably, after completing the forward propagation of the current text segment, the model performs forward propagation on the next text segment until all text segments of the target text sequence have completed forward propagation. Each text segment of the target text sequence undergoes forward propagation sequentially. For text segments with preceding text segments, each attention calculation in the forward propagation can be performed as described in step S220 above. For the first text segment, since there are no preceding text segments, there is no need to load the key-value pair activation values ​​of the preceding text segment; attention calculation is performed directly. Furthermore, after completing the forward propagation of the last text segment, the activation values ​​can be unloaded, and backward propagation of that text segment can be performed directly. Therefore, after all text segments of the target text sequence have completed forward propagation, the backward propagation phase begins.

[0067] Step S230: When performing backpropagation of the model on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation value corresponding to each preceding text segment is loaded in sequence. Attention calculation is performed with the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain the gradient value; the gradient value is unloaded into the central processing unit.

[0068] During the backpropagation phase, when the current text segment Chunk i When performing attention calculations, it is necessary to consider the current text segment chunk during the forward propagation phase. i Q generated during attention calculation i O i The activation value K of the key-value pair with the current text segment and each preceding text segment. j V j The pre-defined backpropagation function (such as the backpropagation function of FlashAttention) is called sequentially for calculation, and Q is obtained for each attention calculation.i K j V j A portion of the gradient, denoted as dQ i,j dK j,i dv j,i Uninstall DK j,i dv j,i On the CPU, with the already stored dK k,i dv k,i Accumulate the sums, where k = j+1, j+2, ..., i. After all attention calculations are complete, set dQ on the CPU. i,j With dQ i,j Accumulate.

[0069] After backpropagation is completed on the current text segment, the next text segment is obtained and backpropagation is performed on it, until all text segments of the target text sequence have completed backpropagation. For each text segment (Chunk)... i Backpropagation is performed sequentially from back to front (i.e., from i=n to i=1). Specifically, when backpropagating the first text segment Chunk1, it is unnecessary to load the aforementioned key-value pair activation values ​​again. After all text segments have completed forward propagation, backpropagation can be performed on each text segment sequentially until all text segments have completed backpropagation.

[0070] In related technologies, to train large language models on text, long sequences of activation values ​​are split into multiple short sequences and placed on different GPUs. Model training is then completed through the computation and communication of multiple GPUs. Alternatively, activation value recomputation techniques are used to increase computational requirements while reducing GPU memory requirements for model training. However, none of these methods can improve the execution efficiency of text training tasks for large language models under GPU resource constraints.

[0071] In this embodiment, the target text sequence is segmented and trained using CPU memory offloading technology to train a large language model. The original target text sequence is divided into multiple segments, which are then sequentially fed into the large language model for processing. Only one segment is forwarded or backwarded at a time, and the GPU stores the activation values ​​of only one segment. This avoids the high GPU memory requirements associated with complete text sequences, thus overcoming the GPU memory bottleneck for model training when the target text sequence is long. Compared to distributed training methods in related technologies, this embodiment does not rely on a large amount of GPU resources, but instead utilizes lower-cost CPU memory resources for training, thereby reducing hardware costs and barriers to entry. Furthermore, this embodiment performs key-value pair activation value loading and unloading simultaneously with attention calculation, overlapping transmission and computation time overhead for higher overall efficiency. Therefore, this embodiment can improve the execution efficiency of text training tasks for large language models even with limited GPU resources.

[0072] Therefore, through the above steps S210 to S230, the current text segment is obtained; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in sequence; when performing forward propagation of the model on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are loaded sequentially from the central processing unit for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit; when performing backward propagation of the model on the current text segment, if the current text segment has a preceding text segment, During attention calculation, the forward propagation output and key-value pair activation values ​​corresponding to the current text segment are loaded from the central processing unit (CPU). Then, the attention activation values ​​corresponding to each preceding text segment are loaded sequentially. Attention calculation is performed on these values ​​along with the forward propagation output and key-value pair activation values ​​corresponding to the current text segment to obtain the gradient value. The gradient value is then unloaded into the CPU. This approach can overcome the GPU memory bottleneck in text sequence training and overlap the transmission time overhead with the computation time overhead, thereby improving the execution efficiency of text training tasks for large language models under GPU resource constraints.

[0073] In another embodiment, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation. Specifically, this may include:

[0074] If the current text segment has at least two preceding text segments, then all preceding text segments except the first text segment are taken as target preceding text segments in sequence; while performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the previous text segment of the target preceding text segment is loaded; after completing the attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the target preceding text segment is deleted.

[0075] In cases where the current text segment has multiple preceding text segments, attention calculations can be performed sequentially based on the key-value pair activation values ​​corresponding to each preceding text segment. That is, if the current text segment has multiple preceding text segments, multiple attention calculations need to be performed on that current text segment. During the forward propagation phase, in the multiple attention calculations for the same text segment, except for the last attention calculation, the key-value pair activation values ​​of the preceding text segments can be loaded simultaneously with each attention calculation, ensuring that the loading of key-value pair activation values ​​is synchronized with the attention calculation.

[0076] For example, if the target text sequence is divided into four segments, and the current segment is the fourth segment, and attention calculation is being performed based on the key-value (KV) activation values ​​corresponding to the third segment, the KV activation values ​​corresponding to the second segment can be loaded from the CPU into the GPU memory first. After completing the attention calculation based on the KV activation values ​​of the third segment, attention calculation is performed again based on the KV activation values ​​of the second segment, and the KV activation values ​​corresponding to the second segment are deleted from the GPU memory after this attention calculation is completed. Simultaneously, while performing attention calculation based on the KV activation values ​​of the second segment, the KV activation values ​​of the first segment are loaded into the GPU memory.

[0077] Therefore, in this embodiment, attention calculation and KV activation value transmission are performed synchronously, thereby reducing the overall time overhead in the forward propagation process and improving the efficiency of forward propagation.

[0078] In another embodiment, when performing model forward propagation on the current text segment, the above model training task execution method may further include:

[0079] If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0080] When performing forward propagation of the model on the current text segment, both attention and non-attention computations are included. For example, Embedding, Dropout, LayerNorm, and QKV projection are all non-attention computations. During the non-attention computation phase, activation values ​​other than KV activation values ​​can be used as target activation values ​​and offloaded to the CPU.

[0081] For example, for the current text segment Chunk i Each non-attention calculation is performed sequentially, and the resulting activation values, excluding Ki and Vi, are marked as... If the current text segment Chunk i If it is not the first text segment Chunk1, then during non-attention calculation, remove the target activation values ​​other than the KV activation values ​​corresponding to the previous text segment. .

[0082] Additionally, after completing the attention calculation for a specific decoder block in the model, the non-attention calculations performed sequentially may include: attention calculation output projection, random deactivation (Dropout), ResidualAdd, LayerNorm, Feedforward Neural Network (FFN) calculation, Dropout, ResidualAdd, and LayerNorm and QKV projection for the next decoder block, until the next attention calculation is encountered or model training ends. The output during the attention calculation phase will be... i And all target activation values ​​other than KV activation values ​​generated in the current non-attention computation are collectively labeled as If the currently processed text segment is not the first text segment Chunk1, then while performing the current non-attention calculation, unload the previous text segment Chunk. i-1 The corresponding target activation value Furthermore, the attention and non-attention calculations described above can be repeated for each decoder block in the model until each decoder block completes its calculation.

[0083] Therefore, in this embodiment, asynchronous unloading of different types of activation values ​​is implemented, and the data transmission of the unloaded data is carried out simultaneously with the computation of the model, thereby masking the time overhead of data transmission. When the forward propagation of the i-th text segment is performed, the non-KV activation values ​​of the (i-1)-th text segment are unloaded to CPU memory; when the backward propagation of the i-th text segment is performed, the (i-1)-th text segment can be reloaded to the GPU for use in the backward propagation of the (i-1)-th segment. The activation values ​​of the last text segment are neither unloaded nor loaded.

[0084] During attention computation, the query vector of the current text segment is paired sequentially with the key-value (KV) activation values ​​of each preceding text segment to perform partial attention computation. Simultaneously, the KV activation values ​​generated for the current text segment are asynchronously unloaded. The unloading of non-KV activation values ​​occurs during non-attention computation time. The unloading / loading of KV activation values ​​and non-KV activation values ​​are interleaved, achieving deep overlap between transmission and computation. This scheduling mechanism ensures that the high-speed Peripheral Component Interconnect Express (PCIe) bandwidth is not simultaneously monopolized by two activation types, improving input / output (I / O) utilization.

[0085] This embodiment implements different scheduling priorities and timings for key-value (KV) activation values ​​and non-KV activation values. Given the higher time sensitivity of KV activation values ​​during cross-segment attention calculations, loading and unloading occur during attention calculations. The transmission of KV activation values ​​is considered high priority, while the transmission of non-KV activation values ​​is set low priority and performed during non-attention calculations. This allows for phased transmission without blocking KV activation value transmission. This hierarchical and refined activation value unloading scheduling strategy further improves the concurrency efficiency of data transmission and model computation.

[0086] In another embodiment, when performing backpropagation of the model on the current text segment, the above-mentioned model training task execution method may further include:

[0087] If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0088] During backpropagation, the current text segment Chunk i The non-attention computation of the model's backpropagation is performed from back to front until the next attention computation is needed, or the backpropagation ends. If the currently processed text segment is not the first text segment Chunk1, the previous text segment Chunk1 is loaded during the non-attention computation. i-1 The target activation value generated during this non-attention computation step is sent to the GPU.

[0089] Understandably, during backpropagation attention calculations, the key-value (KV) activation values ​​of the preceding text segments that haven't yet been calculated are loaded into the GPU for use in the next attention calculation. For example, while performing attention calculations based on the KV activation values ​​of the third text segment, the KV activation values ​​of the second text segment are loaded from the CPU into the GPU.

[0090] In this way, during the reverse propagation process, the unloading and loading of KV activation values ​​and non-KV activation values ​​can be interspersed, thereby avoiding the simultaneous appropriation of PCIe bandwidth by different types of activation values ​​and improving I / O utilization.

[0091] In another embodiment, after backpropagation has been completed for all text segments of the target text sequence, the above model training task execution method may further include:

[0092] Based on the backpropagation results of all text segments of the target text sequence, the model parameters and the optimizer state of the trained model are updated.

[0093] Understandably, backpropagation is performed sequentially on each text segment of the target text sequence, from back to front, until backpropagation is complete. After backpropagation has been completed on all text segments, the model parameters and optimizer state of the trained model are updated based on the backpropagation results stored in the CPU. The model is trained iteratively on the target text sequence until a preset training termination condition is reached. This preset training termination condition could be that the model output reaches a preset standard, or that a preset number of training iterations has been reached, etc.; this embodiment does not specifically limit this.

[0094] Based on this, this embodiment can overcome the GPU memory bottleneck in text sequence training when performing model training tasks, without relying on a large amount of GPU resources, thus reducing the hardware cost and threshold of training. Furthermore, it can improve the efficiency of training tasks even when GPU resources are limited.

[0095] In this embodiment, a model training task execution method is also provided for a central processing unit. Figure 4 This is a flowchart of the model training task execution method in this embodiment, as follows: Figure 4 As shown, it includes the following steps:

[0096] Step S401: In response to the graphics processor performing attention calculations in the forward propagation of the model on the current text segment, the graphics processor sequentially returns each first read value to the graphics processor for the first loading operation of each first read value; and receives and stores the key-value pair activation value corresponding to the current text segment unloaded by the graphics processor; wherein each first read value is the key-value pair activation value corresponding to each preceding text segment of the current text segment; the first loading operation is initiated by the graphics processor when it determines that the current text segment has a preceding text segment; the current text segment is: one of the text segments after the target text sequence is segmented in sequence.

[0097] Step S402: In response to the graphics processor performing attention calculations for backpropagation of the model on the current text segment, a second read value is returned to the graphics processor for the second loading operation of the second read value; and in response to the graphics processor performing third loading operations for each first read value in sequence, each first read value is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and key-value pair activation value corresponding to the current text segment; the gradient value unloaded by the graphics processor is received and stored; the gradient value is obtained by performing attention calculations based on the second read value and each first read value respectively.

[0098] Steps S401 to S402 described above can overcome the GPU memory bottleneck in text sequence training and overlap the transmission time overhead with the computation time overhead, thereby improving the execution efficiency of text training tasks for large language models under GPU resource constraints. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0099] In one embodiment, the above model training task execution method may further include, prior to the graphics processor performing the forward propagation of the model:

[0100] Based on the model initialization parameters corresponding to the model to be trained, determine the maximum segment size that the graphics processor can accommodate; based on the maximum segment size, segment the target text sequence to obtain each text segment.

[0101] When segmenting a target text sequence, the maximum segment size that the GPU memory can accommodate can be estimated by referring to the model's initialization parameters. This is used to segment the target text sequence. The segment size estimation formula can be set according to the model architecture. For example, for the GPT model, the corresponding estimation formula can be:

[0102] ;

[0103] Where b, h, L, V, num_param, and gpu_mem represent the training batch size, model hidden dimension size, model number of layers, model vocabulary size, model parameter count, and GPU memory capacity, respectively.

[0104] In this embodiment, segmented planning can be implemented based on model characteristics and architecture, thereby further improving the efficiency of model training task execution.

[0105] In one embodiment, the above-mentioned model training task execution method may further include:

[0106] Based on the model initialization parameters, the length of the target text sequence, and the maximum segment size, determine the required memory capacity of the central processing unit; when the required memory capacity is greater than the available memory capacity of the central processing unit, set a recalculation strategy for the model to be trained.

[0107] The amount of CPU memory required for training can be estimated based on the model initialization parameters, the length of the target text sequence, and the maximum segment size that the GPU memory can hold. For example, for the GPT model, it can be calculated as follows:

[0108] ;

[0109] Here, s0 represents the maximum sequence length required for training, i.e., the length of the target text sequence; cpu_mem0 represents the CPU memory required for training. If the calculated cpu_mem0 exceeds the available CPU memory capacity, a recalculation strategy is applied to the activation functions and layer normalization modules in the model to be trained, recalculating the required CPU memory size and the maximum segment size that GPU memory can accommodate. If the required CPU memory size still exceeds the available CPU memory capacity, a further recalculation strategy is applied to the linear layers in the model, recalculating the required CPU memory size and the maximum segment size that GPU memory can accommodate again. Recalculation can specifically be a memory optimization technique in deep learning, using a time-for-space tradeoff to reduce the model's memory usage during training.

[0110] In addition, the computation time of each module can be determined by performing single-module tests on each module of the model to be trained, and the overall computation time under different recomputation strategies can be estimated by multiplying the computation time required for a single text segment by the number of segments, so as to select the recomputation strategy with the shortest computation time required for training.

[0111] In this embodiment, considering the potential CPU memory shortage after offloading the activation values ​​of each text segment to the CPU, which could limit the expansion of the trainable text sequence length, this embodiment adaptively selects a subset of modules in the model to apply a recomputation strategy when CPU memory is insufficient. This allows for training longer text sequences without increasing CPU memory. Specifically, modules with lower computational overhead for recomputation can be prioritized to avoid significantly impacting the overall efficiency of the training task.

[0112] Furthermore, this embodiment also considers that in some cases, recomputation can allow for larger segment sizes, thereby improving the utilization of GPU computing units and covering the negative impact of recomputation itself on computational efficiency. Therefore, for different recomputation strategies, not only the capacity of GPU memory and CPU memory are considered, but also the computation time. The computation time of each module is obtained through single-module testing, and combined with the activation value occupancy of each module, the usable and most efficient recomputation strategy and segment size selection are obtained.

[0113] Therefore, this embodiment utilizes CPU memory as a cache space for activation values, and all activation values ​​generated during the process are segmented and unloaded along the sequence dimension. Compared to related technologies, this embodiment can support training longer text sequences when CPU memory resources are sufficient. Furthermore, a selective recomputation strategy is used to alleviate the problem of limited CPU memory, and the strategy is adaptively adjusted through single-module testing to find the optimal balance between computational efficiency and long sequence scalability.

[0114] It should also be noted that before training, the model structure of the large language model to be trained (such as GPT, Llama, GLM) can be packaged. Specifically, based on the attention calculation modules (Attn) in the large language model... i Where i = 1, 2, ..., L, L is the structural layer number of the large language model), the large language model is divided into L attention computation blocks (Attn) at their positions. i ), and L+1 non-attention computation blocks (Not_Attn) i ).

[0115] Figure 5 This is a schematic diagram illustrating the structural segmentation of a large language model in this embodiment. For example... Figure 5 As shown, the large language model to be trained can be divided into several attention computation blocks: attention computation block 1 (Attn1), attention computation block 2 (Attn2), etc.; and several non-attention computation blocks: non-attention computation block 1 (Not_Attn1), non-attention computation block 2 (Not_Attn2), non-attention computation block 3 (Not_Attn3) up to non-attention computation block L+1 (Not_Attn). L+1Within each attention calculation block, attention calculations are performed, such as self-attention. Within each non-attention calculation block, non-attention calculations are performed, such as embedding, QKV projection (OKVproj), linear calculations (including Linear1 and Linear2), dropout, layer normalization, Gaussian error linearization (GeLU), and loss calculation.

[0116] Figure 6 This is a schematic diagram illustrating one type of activation value transmission in this embodiment. For example... Figure 6 As shown, the target text sequence is divided into four segments: text segment 1, text segment 2, text segment 3, and text segment 4. During the forward propagation phase, for each layer in the model ( Figure 6 When layers 1, 2, and 3 in the process perform calculations, during the attention calculation of text segment 1, key-value (KV) activation values ​​are transferred (KV activation values ​​are unloaded or loaded). Figure 6 (The yellow squares represent KV activation values). During the non-attention calculation of text segment 2, the non-KV activation values ​​of text segment 1 are then transferred (during forward propagation, this is to unload the non-KV activation values). Figure 6 (Non-key-value activation values ​​are represented by blue squares in the text). Similarly, when performing attention calculation on text segment 2, key-value activation values ​​are simultaneously transmitted, and the same applies to text segments 3 and 4. During the backpropagation phase, when performing attention calculation on text segment 4, key-value activation values ​​are loaded simultaneously; when performing non-attention calculation on text segment 3, non-key-value activation values ​​of text segment 4 are loaded, and so on. In both forward and backward propagation, attention calculation is achieved by transmitting key-value activation values ​​multiple times, and non-key-value activation values ​​from the previous text segment are asynchronously unloaded. Therefore, a hierarchical and fine-grained unloading scheduling strategy is implemented.

[0117] Figure 7 This is a schematic diagram of a partial structure of a large language model in this embodiment. For example... Figure 7As shown, a large language model can include L layers, each of which can include an attention module and a feedforward neural network (FFN) module. The attention module includes QKV projection computation (QKV proj), self-attention computation, linear computation, and dropout computation. The FFN module includes linear computation 1 (Linear1), Gaussian error linear computation (GeLU), linear computation 2 (Linear1), and dropout computation. Furthermore, layer normalization (LayerNorm) is included between the attention and FFN modules. The large language model can also include embedding computation and loss calculation.

[0118] Figure 8 This is a schematic diagram illustrating a combination of segmented calculation and activation value transmission in this embodiment, as shown below. Figure 8 As shown, if the target text sequence is divided into 4 text segments, then the attention calculation is performed sequentially for text segment 1, text segment 2, text segment 3, and text segment 4, according to the segmentation of the target text sequence. In each attention calculation (specifically as follows...) Figure 5In the self-attention calculation process: When calculating the attention for text segment 1 (calculating the query vector Q1 with the corresponding KV activation value KV1 of the text segment) to obtain the result O_11, the KV activation value KV1 of text segment 1 is offloaded to the CPU. When calculating the attention for text segment 2, it is necessary to first perform an attention calculation based on the query vector Q2 corresponding to text segment 2 and the KV activation value KV1 corresponding to text segment 1, obtaining the result O_21; then, a second attention calculation is performed based on the KV activation value KV2 corresponding to text segment 2 and the query vector Q2 to obtain the result O_22. During the second attention calculation, the KV activation value KV2 of text segment 2 is offloaded to the CPU. When calculating the attention for text segment 3, the KV activation values ​​KV1 (corresponding to text segment 1), KV2 (corresponding to text segment 2), and KV3 (corresponding to text segment 3) are sequentially combined with the query vector Q3 corresponding to the text segment to perform the attention calculation, outputting the corresponding calculation results O_31, O_32, and O_33 respectively. Similarly, when calculating the attention for text segment 4, the KV activation values ​​KV1 (corresponding to text segment 1), KV2 (corresponding to text segment 2), KV3 (corresponding to text segment 3), and KV4 (corresponding to text segment 4) are sequentially combined with the query vector Q4 corresponding to the text segment to perform the attention calculation, outputting the corresponding calculation results O_41, O_42, O_43, and O_44 respectively. Furthermore, the corresponding KV activation values ​​can be unloaded and loaded during each attention calculation.

[0119] Additionally, during non-attentional computation (linear computation, layer normalization, activation functions, random deactivation, residual connections, etc.), non-KV activation values ​​are unloaded into the CPU. Based on the attention and non-attentional computation results (non-attentional computation result 1, non-attentional computation result 2, non-attentional computation result 3, non-attentional computation result 4), the training outputs corresponding to each text segment are obtained (training output 1, training output 2, training output 3, training output 4).

[0120] Figure 9 This is a flowchart of a model training task execution method for some embodiments. This method is used to train a GPU-1.3B model on a configuration of a single H20 GPU with 96GB of video memory and a CPU with 250GB of RAM, using a target text sequence of length 128K (a single sequence has 128×1024 tokens). Figure 9 As shown, the method for performing the model training task includes the following steps:

[0121] Step S901, model initialization; wherein, the model can be initialized using the parameter settings of GPT-1.3B, and in accordance with Figure 4 The partitioning shown replaces and repackages the model structure of GPT-1.3B. Specifically, since GPT-1.3B has 24 layers, the model is divided into 24 attention computation segments and 25 non-attention computation segments.

[0122] Step S902 involves determining the segmentation and recomputation strategies for the model. The GPT-1.3B model and optimizer state require approximately 1.3 × 16 = 20.8 GB of GPU memory, leaving about 96 GB - 20.8 GB = 75.2 GB of available GPU memory to store segmented activation values. The GPU memory and CPU memory requirements for the activation values ​​are determined, and it is found that CPU memory is sufficient for training. Furthermore, applying recomputation to the activation function and layer normalization model increases the segment size of the text to 32K, and this setting demonstrates the highest overall efficiency in single-module testing.

[0123] Step S903: Segment the target text sequence; the length of the input target text sequence is 128K, and the target text sequence is divided into 4 segments, each segment being 32K in length.

[0124] In step S904, during the forward propagation phase, each text segment is processed sequentially, while the non-KV activation values ​​of the previous text segment are asynchronously unloaded into the CPU. The specific transmission of KV and non-KV activation values, as well as the execution mechanisms for attention and non-attention calculations, can be found in the above embodiments and will not be repeated here.

[0125] Step S905: During the backpropagation phase, each text segment is processed sequentially, while non-KV activation values ​​are asynchronously loaded from the CPU to the GPU. Specifically, the attention calculation process entirely handles the unloading and loading of KV activation values, without processing the transmission of non-KV activation values. For details, please refer to the above embodiment; further elaboration is unnecessary.

[0126] Step S906: Repeat steps S904 and S905 until the model converges or reaches the predetermined number of training rounds, then end the training.

[0127] Steps S901 to S906 above overcome the GPU memory bottleneck in text sequence training by utilizing the asynchronous transmission mechanism of text segmentation calculation and activation values. Compared to existing distributed training methods, this embodiment does not rely on a large amount of GPU resources, but uses relatively low-cost CPU memory resources, reducing training costs and barriers. The asynchronous transmission scheduling mechanism overlaps the transmission time overhead with the computation time overhead. Compared to related technologies, this avoids additional computational overhead and achieves higher overall efficiency. Furthermore, by using CPU memory as a cache space for activation values, all activation values ​​generated during training are segmented and unloaded along the sequence dimension. In scenarios with sufficient CPU memory resources, this allows for training longer text sequence lengths compared to existing related technologies.

[0128] This embodiment also provides a graphics processor for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0129] Figure 10 This is a structural block diagram of the graphics processor 10 in this embodiment, as shown below. Figure 10 As shown, the graphics processor 10 includes: an acquisition module 12, a forward propagation module 14, and a backward propagation module 16;

[0130] The acquisition module 12 is used to acquire the current text segment; wherein the current text segment is one of the text segments after the target text sequence is segmented in order; the forward propagation module 14 is used to, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, load the key-value pair activation values ​​corresponding to each preceding text segment from the central processing unit and perform attention calculation; at the same time, unload the key-value pair activation values ​​corresponding to the current text segment to the central processing unit; the back propagation module 16 is used to, when performing model back propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, load the forward propagation output result and key-value pair activation values ​​corresponding to the current text segment from the central processing unit, and load the attention activation values ​​corresponding to each preceding text segment in order, perform attention calculation with the forward propagation output result and key-value pair activation values ​​corresponding to the current text segment to obtain the gradient value; unload the gradient value to the central processing unit.

[0131] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0132] The aforementioned graphics processor 10 can overcome the GPU memory bottleneck in text sequence training and overlap the transmission time overhead with the computation time overhead, thereby improving the execution efficiency of text training tasks for large language models under the condition of limited GPU resources.

[0133] In one embodiment, the forward propagation module 14 is specifically configured to: if the current text segment has at least two preceding text segments, sequentially take all preceding text segments other than the first text segment as target preceding text segments; while performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, load the key-value pair activation value corresponding to the previous text segment of the target preceding text segment; and after completing the attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, delete the key-value pair activation value corresponding to the target preceding text segment.

[0134] In one embodiment, the forward propagation module 14 is further configured to, if the current text segment has a preceding text segment, unload the target activation value corresponding to the previous text segment of the current text segment to the central processing unit when performing non-attention calculation on the current text segment; the target activation value is an activation value other than the key-value pair activation value.

[0135] In one embodiment, the backpropagation module 16 is further configured to load the target activation value corresponding to the previous text segment of the current text segment from the central processing unit when performing non-attention calculation on the current text segment if the current text segment has a preceding text segment; the target activation value is an activation value other than the key-value pair activation value.

[0136] In one embodiment, the graphics processor 10 further includes an update module for updating the model parameters and the optimizer state of the trained model based on the backpropagation results of all text segments of the target text sequence.

[0137] This embodiment also provides a model training system. Figure 11 This is a schematic diagram of the model training system 11 in this embodiment, as shown below. Figure 11 As shown, the model training system 11 includes a graphics processor 112 and a central processing unit 114; wherein: the graphics processor 112 is used to execute the model training task execution method for the graphics processor provided in any of the above embodiments; and the central processing unit 114 is used to execute the model training task execution method for the central processing unit provided in any of the above embodiments.

[0138] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0139] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0141] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0142] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for executing a model training task, characterized in that, For a graphics processor, the method includes: Obtain the current text segment; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in order; When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit (CPU) for attention calculation; simultaneously, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the CPU; the preceding text segment of the current text segment is: the text segment that precedes the current text segment when the target text sequence is segmented sequentially; When performing backpropagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the forward propagation output and key-value pair activation values ​​corresponding to the current text segment are loaded from the central processing unit, and the attention activation values ​​corresponding to each preceding text segment are loaded sequentially. Attention calculation is then performed with the forward propagation output and key-value pair activation values ​​corresponding to the current text segment to obtain gradient values; the gradient values ​​are then unloaded into the central processing unit.

2. The model training task execution method according to claim 1, characterized in that, When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then during attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation, including: If the current text segment has at least two preceding text segments, then all preceding text segments other than the first text segment are taken as the target preceding text segments in sequence; While performing attention calculation based on the activation value of the key-value pair corresponding to the target preceding text segment, the activation value of the key-value pair corresponding to the previous text segment of the target preceding text segment is loaded; After performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, delete the key-value pair activation value corresponding to the target preceding text segment.

3. The model training task execution method according to claim 1, characterized in that, When performing model forward propagation on the current text segment, the method further includes: If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

4. The model training task execution method according to claim 1, characterized in that, When performing backpropagation of the model on the current text segment, the method further includes: If the current text segment has a preceding text segment, then when performing non-attention computation on the current text segment, the target activation value corresponding to the preceding text segment of the current text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

5. The method for executing a model training task according to any one of claims 1 to 4, characterized in that, After all text segments of the target text sequence have completed backpropagation, the method further includes: Based on the backpropagation results of all text segments of the target text sequence, the model parameters of the trained model and the optimizer state of the model are updated.

6. A method for executing a model training task, characterized in that, For a central processing unit, the method includes: In response to the graphics processor performing attention calculations during model forward propagation on the current text segment, the system sequentially returns each first read value to the graphics processor for each first read value in a first loading operation; and receives and stores the key-value pair activation values ​​corresponding to the current text segment unloaded by the graphics processor; wherein each first read value is a key-value pair activation value corresponding to each preceding text segment of the current text segment; the first loading operation is initiated by the graphics processor when it determines that the current text segment has a preceding text segment; the current text segment is one of the text segments after the target text sequence is segmented sequentially; the preceding text segment of the current text segment is the text segment ordered before the current text segment when the target text sequence is segmented sequentially. In response to the graphics processor performing attention calculations for backpropagation of the model on the current text segment, the second read value is returned to the graphics processor for a second loading operation on the second read value; and in response to the graphics processor performing third loading operations on each of the first read values ​​in sequence, each of the first read values ​​is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and key-value pair activation value corresponding to the current text segment; the gradient value unloaded by the graphics processor is received and stored; the gradient value is obtained by performing attention calculations based on the second read value and each of the first read values ​​respectively.

7. The model training task execution method according to claim 6, characterized in that, Before the graphics processor performs the forward propagation of the model, the method further includes: The maximum segment size that the graphics processor can accommodate is determined based on the model initialization parameters corresponding to the model to be trained; Based on the maximum segment size, the target text sequence is segmented to obtain each text segment.

8. The model training task execution method according to claim 7, characterized in that, The method further includes: The required memory capacity of the central processing unit is determined based on the model initialization parameters, the length of the target text sequence, and the maximum segment size. When the required memory capacity is greater than the available memory capacity of the central processing unit, a recalculation strategy is set for the model to be trained.

9. A graphics processor, characterized in that, include: The module consists of an acquisition module, a forward propagation module, and a back propagation module. The acquisition module is used to acquire the current text segment; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in sequence; The forward propagation module is configured to, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, load the key-value pair activation values ​​corresponding to each preceding text segment sequentially from the central processing unit (CPU) for attention calculation; simultaneously, unload the key-value pair activation values ​​corresponding to the current text segment to the CPU; the preceding text segment of the current text segment is: the text segment ordered before the current text segment when the target text sequence is segmented sequentially; The backpropagation module is configured to, when performing model backpropagation on the current text segment, if the current text segment has a preceding text segment, load the forward propagation output result and key-value pair activation value corresponding to the current text segment from the central processing unit during attention calculation, and sequentially load the attention activation value corresponding to each preceding text segment, perform attention calculation with the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain a gradient value; and unload the gradient value to the central processing unit.

10. A model training system, characterized in that, include: Graphics processor and central processing unit; The graphics processor is used to execute the model training task execution method according to any one of claims 1 to 5; The central processing unit is used to execute the model training task execution method according to any one of claims 6 to 8.

Images