Model training task execution method, graphics processor and model training system

Abstract

The invention relates to a model training task execution method, a graphics processing unit and a model training system.The model training task execution method comprises the steps that when attention calculation of model forward propagation is executed on a current text segment, the current text segment is subjected to attention calculation; sequentially loading key values corresponding to all the preamble segments from the central processing unit to carry out attention calculation on the activation values; meanwhile, the key value pair activation value corresponding to the current text segment is unloaded to the central processing unit; when the attention calculation of model back propagation is executed on the current text segment, loading a forward propagation output result and a key value pair activation value corresponding to the current text segment from a central processing unit, and loading the attention activation value corresponding to each preorder text segment in sequence; performing attention calculation on the activation value according to the forward propagation output result and the key value corresponding to the current text segment to obtain a gradient value; unloading the gradient value to a central processing unit; under the condition that GPU resources are limited, the execution efficiency of a text training task of a large language model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training task execution method, a graphics processor, and a model training system. Background Art

[0002] To improve the understanding and processing capabilities of large language models such as the Generative Pre-trained Transformer (GPT), the AI ​​Large Language Model (Llama), and the General Language Model (GLM) for longer texts, such as entire books and contextual conversations with more than 32K tokens, it is necessary to input such longer texts (hereinafter referred to as long sequences) into the large language models for training.

[0003] The training process involves a cyclical process of text input, model calculation, and model parameter updates. Model calculation is divided into two steps: forward propagation and backward propagation. During the forward propagation process, each module of the large language model sequentially calculates outputs based on the input text content. These outputs, called activation values, need to be retained. During the backward propagation process, further calculations are performed based on these activation values, and the model parameters of the large language model are updated. Therefore, it can be seen that the activation values ​​generated during forward propagation need to be stored in the graphics processing unit (GPU) memory, and the space occupied by these activation values ​​will increase with the length of the text. When the length of the text input to the large language model reaches tens of thousands or even millions of tokens, the GPU memory required for the activation values ​​generated by training will exceed the memory capacity of a single GPU.

[0004] In some related technologies, a distributed sequence parallel training method is often used to divide the activation values ​​of a long sequence into multiple short sequences, which are placed on multiple GPUs on a single server, or on multiple GPUs on multiple servers in a cluster. Multiple GPUs exchange data and perform parallel calculations. This method requires a large number of GPUs to maintain training, and multiple GPUs and even multiple servers need to communicate. Therefore, the hardware and communication requirements are large, and it is impossible to achieve efficient large language model training under limited GPU resources. In other related technologies, activation value recalculation technology is used to reduce the graphics memory requirements for model training by increasing the computing requirements. The overall efficiency of this type of method is reduced by the additional computing requirements of activation value recalculation, and the recalculation technology still needs to retain complete activation values ​​for some model modules. As the sequence length increases, the graphics memory usage will still increase. Therefore, in the current related technologies, the execution efficiency of text training tasks for large language models under limited GPU resources is low.

[0005] Therefore, there is currently no effective solution to the problem in related technologies of low efficiency in executing text training tasks for large language models when GPU resources are limited. Summary of the Invention

[0006] In this embodiment, a model training task execution method, a graphics processor, and a model training system are provided to solve the problem of low execution efficiency of text training tasks for large language models when GPU resources are limited in the related art.

[0007] In a first aspect, this embodiment provides a method for executing a model training task, which is used for a graphics processor. The method includes:

[0008] Obtaining a current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence;

[0009] When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each of the preceding text segments are sequentially loaded from the central processing unit for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit;

[0010] When performing model back propagation on the current text segment, if there is a preceding text segment for the current text segment, then when performing attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation value corresponding to each of the preceding text segments is loaded in turn, and attention calculation is performed on the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain a gradient value; and the gradient value is unloaded to the central processing unit.

[0011] In some embodiments, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit to perform attention calculation, including:

[0012] If the current text segment has at least two preceding text segments, all preceding text segments except the first text segment are sequentially used as target preceding text segments;

[0013] While performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, loading the key-value pair activation value corresponding to the previous text segment of the target preceding text segment;

[0014] After completing the attention calculation according to the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the target preceding text segment is deleted.

[0015] In some embodiments, when performing model forward propagation on the current text segment, the method further includes:

[0016] If the current text segment has a preceding text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0017] In some embodiments, when performing model back propagation on the current text segment, the method further includes:

[0018] If the current text segment has a preceding text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0019] In some embodiments, after backpropagation is completed for all text segments of the target text sequence, the method further includes:

[0020] Based on the back-propagation results of all text segments of the target text sequence, the model parameters of the trained model and the optimizer state of the model are updated.

[0021] In a second aspect, a method for executing a model training task is provided in this embodiment, for use in a central processing unit. The method includes:

[0022] In response to the graphics processor performing the attention calculation in the forward propagation of the model on the current text segment, the graphics processor sequentially performs the first loading operation on each first read value, and returns each first read value to the graphics processor respectively; and receives and stores the key-value pair activation value corresponding to the current text segment unloaded by the graphics processor; wherein each first read value is the key-value pair activation value corresponding to each preceding text segment of the current text segment; the first loading operation is initiated by the graphics processor when it is determined that the current text segment has a preceding text segment; the current text segment is: one of the text segments after the target text sequence is sequentially segmented;

[0023] In response to the second loading operation for the second read value when the graphics processor performs the attention calculation of the model back propagation on the current text segment, the second read value is returned to the graphics processor; and in response to the third loading operation of the graphics processor for each first read value in sequence, each first read value is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and key-value pair activation value corresponding to the current text segment; receiving and storing the gradient value unloaded by the graphics processor; the gradient value is obtained by performing the attention calculation based on the second read value and each first read value respectively.

[0024] In some embodiments, before the graphics processor performs forward propagation of the model, the method further includes:

[0025] Determining a maximum segment size that can be accommodated by the graphics processor based on model initialization parameters corresponding to the model to be trained;

[0026] According to the maximum segment size, the target text sequence is segmented to obtain text segments.

[0027] In some embodiments, the method further comprises:

[0028] Determining the required memory capacity of the central processing unit according to the model initialization parameters, the target text sequence length and the maximum segment size;

[0029] When the required memory capacity is greater than the available memory capacity of the central processing unit, a recalculation strategy is set for the model to be trained.

[0030] In a third aspect, a graphics processor is provided in this embodiment, including: an acquisition module, a forward propagation module, and a backward propagation module;

[0031] The acquisition module is used to acquire a current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence;

[0032] The forward propagation module is configured to, when performing model forward propagation on the current text segment, if there is a previous text segment for the current text segment, sequentially load the key-value pair activation values ​​corresponding to each of the previous text segments from the central processing unit to perform attention calculation; and simultaneously, unload the key-value pair activation values ​​corresponding to the current text segment to the central processing unit;

[0033] The back propagation module is used to, when performing model back propagation on the current text segment, if the current text segment has a preceding text segment, load the forward propagation output result and key-value pair activation value corresponding to the current text segment from the central processing unit when performing attention calculation, and sequentially load the attention activation value corresponding to each of the preceding text segments, perform attention calculation with the forward propagation output result and key-value pair activation value corresponding to the current text segment, and obtain a gradient value; and unload the gradient value to the central processing unit.

[0034] Fourthly, in this embodiment, a model training system is provided, including: a graphics processor and a central processing unit;

[0035] The graphics processor is used to execute the model training task execution method described in the first aspect above;

[0036] The central processing unit is used to execute the model training task execution method described in the second aspect above.

[0037] Compared with related technologies, this embodiment provides a model training task execution method, a graphics processor, and a model training system. The model training task execution method obtains the current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence; when executing the model forward propagation on the current text segment, if the current text segment has a previous text segment, then when executing the attention calculation, the key-value pair activation values ​​corresponding to each previous text segment are loaded from the central processing unit in sequence for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit; when executing the model backward propagation on the current text segment, if the current text segment has a previous text segment, then when executing the attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation value corresponding to each previous text segment is loaded in sequence, and the attention calculation is performed with the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain the gradient value; the gradient value is unloaded to the central processing unit; it can break through the GPU memory bottleneck of text sequence training and overlap the transmission time overhead with the calculation time overhead, thereby achieving the improvement of the execution efficiency of the text training task of the large language model under the condition of limited GPU resources.

[0038] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of a terminal of the model training task execution method according to an embodiment of the present application;

[0041] Figure 2 This is a flowchart of a method for executing a model training task according to an embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of the dependency relationship between various text segments during model calculation in an embodiment of the present application;

[0043] Figure 4 This is a flowchart of a method for executing a model training task according to an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the structural segmentation of a large language model in an embodiment of the present application;

[0045] Figure 6 This is a schematic diagram of an activation value transmission method according to an embodiment of the present application;

[0046] Figure 7 This is a schematic diagram of the partial structure of a large language model according to an embodiment of the present application;

[0047] Figure 8 This is a schematic diagram of a combination of segmented calculation and activation value transmission in an embodiment of the present application;

[0048] Figure 9 is a flowchart of a method for executing a model training task in some embodiments of the present application;

[0049] Figure 10 is a structural block diagram of a graphics processor according to an embodiment of the present application;

[0050] Figure 11 It is a structural diagram of the model training system of an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 This is a hardware structure diagram of the terminal of the model training task execution method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model training task execution method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0055] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0056] In this embodiment, a model training task execution method is provided, which is used for a graphics processor. Figure 2 This is a flow chart of the model training task execution method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0057] Step S210 , obtaining a current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence.

[0058] The target text sequence can be a predetermined text sequence that needs to be used to train a large language model. Exemplarily, the target text sequence can be a text sequence with a sequence length exceeding 128K tokens. Before training, the target text sequence can be divided into several text segments in sequence. Specifically, the length of each text segment after segmentation can be made the same, and the length of each text segment can be made a power of 2 (to facilitate GPU calculation and transmission). If the length of the target text sequence itself cannot meet the requirement that the length of each text segment after segmentation is a power of 2, a special token can be added to the end of the target text sequence to make it reach the required length. For example, obtain the target text sequence from the data set, and divide the target text sequence into multiple text segments in sequence without overlap: Chunk1 to Chunk nIn this way, the efficiency of model training can be improved. In addition, it can be understood that the model training task execution method of this embodiment can be applied to a large language model.

[0059] Step S220, when executing the model forward propagation on the current text segment, if the current text segment has a previous text segment, then when performing the attention calculation, the key-value pair activation values ​​corresponding to each previous text segment are loaded from the central processing unit in sequence for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit.

[0060] Forward propagation and backpropagation in large language models are the core processes of neural network training, responsible for calculating predictions and optimizing model parameters, respectively. Forward propagation passes input data through layer-by-layer neural network calculations, ultimately generating a predicted output. Backpropagation adjusts model parameters based on prediction errors to minimize the loss function. Activation values ​​are the intermediate outputs of each layer during forward propagation. Activation gradients are the partial derivatives of the loss function with respect to the activation values ​​of a particular layer during backpropagation.

[0061] After obtaining the above segmentation, each text segment Chunk i (i=1,2,...,n), when entering the forward propagation stage, each text segment Chunk needs to be i (i=1,2,...,n), in the order from front to back (i from 1 to n), the forward propagation process is executed in sequence. i , sequentially perform non-attention calculations of each module in the large language model until the attention calculation module is encountered. Before the attention calculation, the current text segment can be sequentially subjected to word embedding (Embedding), random inactivation (Dropout), layer normalization (LayerNorm), and query-key-value (QKV) projection to obtain the query vector (Q i ), key vector (K i ) and the value vector (V i ), the key vector and value vector constitute the key-value pair activation value (hereinafter referred to as KV activation value). In addition, in addition to the above K i 、V i Other activation values ​​other than , mark the other activation values ​​as .

[0062] The preceding text segment of the current text segment can be: the text segment that is sorted before the current text segment when the target text sequence is segmented in sequence. For example, if the current text segment is Chunk i , then the preceding text segment of the current text segment can include Chunk i-1To Chunk1. Except for the first text segment Chunk1, other text segments (Chunk2 to Chunk n ) has at least one preceding text segment.

[0063] The large language model itself processes the entire target text sequence, so global sequence information is required for calculation during the training process. To this end, the calculation order can be set according to the dependency relationship between each text segment when participating in the model calculation, so as to ensure that the sequential processing of each text segment after segmentation does not affect the model training effect. Figure 3 This is a diagram showing the dependency relationship between various text segments during model calculation. Figure 3 As shown, the calculation of the arrow end point can only be performed after the calculation of the arrow start point is completed. The training process can be divided into two stages: forward propagation and back propagation. The calculations involved in these two stages can be based on Figure 3 The calculations are completed in order as indicated by the arrows in the figure. The calculations may include attention calculations and non-attention calculations. In the forward propagation, the subsequent text segments need the KV activation values ​​calculated by the previous text segments, so from segment 1 (the first text segment) to the last text segment ( Figure 3 In the figure, forward propagation is performed in sequence. In the backward propagation, since the preceding text segment needs the KV activation value gradient calculated by the backward propagation of the subsequent text segment, the backward propagation is performed from the last text segment to the first text segment.

[0064] When performing attention calculation on the current text segment, if the current text segment has a previous text segment, the Q corresponding to the current text segment can be used to calculate the attention. i , and the K corresponding to the current text segment and each preceding text segment of the current text segment j and V j , where j=i,i-1,...1, and sequentially calls the preset forward propagation function (e.g., the forward propagation function based on FlashAttention) to perform attention calculation. While performing attention calculation on the current text segment, the K corresponding to the current text segment can also be unloaded. i 、V i To the Central Processing Unit (CPU). In addition, algorithms such as online softmax can be used to update the results after each attention calculation of the current text segment, and finally obtain the output of the attention calculation O i .

[0065] Calculations on the GPU mainly occupy the computing units in the GPU and do not occupy the transmission bandwidth between the GPU and the CPU, while offloading mainly occupies the transmission bandwidth between the GPU and the CPU. Therefore, by simultaneously performing KV activation value offloading and attention calculation, the computing unit can perform calculations while data is being transmitted, and the total consumed time (data transmission time + calculation time) is closer to the calculation time, thereby masking the time overhead of data transmission. In this embodiment, unloading activation values ​​in the GPU refers to transferring them from the GPU's video memory to the CPU's memory / main memory. Loading activation values ​​from the CPU refers to copying the corresponding activation values ​​from the CPU memory to the GPU. In the GPU, after completing an attention calculation based on the loaded KV activation value, the KV activation value can be directly deleted in the GPU.

[0066] It can be understood that after the forward propagation of the current text segment is completed, the next text segment is obtained to execute the forward propagation of the model until the forward propagation of all text segments of the target text sequence is completed. Among them, the forward propagation is completed in sequence for each text segment of the target text sequence. Among them, for the text segments with a preceding text segment, each attention calculation in the forward propagation can be completed with reference to the above step S220. For the first text segment, since there is no preceding text segment, there is no need to load the activation value of the key-value pair of the preceding text segment, and an attention calculation is performed directly. In addition, after the forward propagation of the last text segment is completed, the activation value does not need to be unloaded, and the reverse propagation of the text segment can be performed directly. Therefore, after all text segments of the target text sequence have completed the forward propagation, the reverse propagation stage is entered.

[0067] Step S230, when performing model back propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation value corresponding to each preceding text segment is loaded in turn, and the attention calculation is performed on the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain the gradient value; the gradient value is unloaded to the central processing unit.

[0068] In the back propagation phase, when the current text segment Chunk i When performing attention calculation, it is necessary to calculate the current text segment Chunk according to the forward propagation stage. i Q generated during attention calculation i , O i , and the key-value pair activation value K of the current text segment and each previous text segment j 、V j , call the preset back propagation function (such as the back propagation function of FlashAttention) in turn to perform calculations, and each attention calculation obtains Qi , K j 、V j Part of the gradient, denoted as dQ i,j 、dK j,i 、dv j,i Uninstall dK j,i 、dv j,i To the CPU, and the stored dK k,i 、dv k,i Accumulate, where k=j+1,j+2,,...i. After the attention calculation is completed, let dQ on the CPU i,j With dQ i,j Accumulate.

[0069] After completing the back propagation for the current text segment, get the next text segment to perform the model back propagation until all text segments of the target text sequence complete the back propagation. i , performing backpropagation sequentially from back to front (i.e., from i=n to i=1). Specifically, when performing backpropagation on the first text segment, Chunk1, there is no need to load the activation values ​​for the key-value pairs described above. After all text segments have completed forward propagation, backpropagation can be performed on each segment in turn until all segments are backpropagated.

[0070] To train large language models on text, conventional techniques involve splitting long sequences of activation values ​​into multiple shorter sequences and placing them on different GPUs. Model training is then accomplished through computation and communication across multiple GPUs. Alternatively, activation recalculation techniques are employed to reduce the GPU memory requirements for model training by increasing computational requirements. None of these approaches, employed in conventional techniques, can improve the efficiency of large language model text training tasks when GPU resources are limited.

[0071] In this embodiment, the target text sequence is segmented and combined with CPU memory unloading technology to achieve text processing training for a large language model. Among them, after the original target text sequence is divided into multiple text segments, they are sequentially sent to the large language model for processing. Only forward propagation or backward propagation of one text segment is performed at the same time, and the GPU only stores the activation value of one text segment, thereby avoiding the high GPU video memory requirements brought by the complete text sequence, and then, when the target text sequence is long, it can break through the GPU video memory bottleneck of model training. Compared with the distributed training method in the related art, this embodiment does not need to rely on more GPU resources, but uses lower-cost CPU memory resources to complete training, thereby reducing the hardware cost and threshold of training. In addition, this embodiment synchronizes the unloading and loading of key-value pair activation values ​​with attention calculations, thereby overlapping the transmission time overhead and the calculation time overhead, thereby achieving higher overall efficiency. Therefore, this embodiment can improve the execution efficiency of text training tasks for large language models when GPU resources are limited.

[0072] Therefore, through the above steps S210 to S230, the current text segment is obtained; wherein, the current text segment is: one of the text segments after the target text sequence is segmented in sequence; when the model forward propagation is performed on the current text segment, if the current text segment has a preceding text segment, then when the attention calculation is performed, the key-value pair activation values ​​corresponding to each preceding text segment are loaded from the central processing unit in sequence to perform attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit; when the model backward propagation is performed on the current text segment, if the current text segment has a preceding text segment, When performing attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation values ​​corresponding to each previous text segment are loaded in turn, and the attention calculation is performed with the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain the gradient value; the gradient value is unloaded to the central processing unit; it can break through the GPU memory bottleneck of text sequence training and overlap the transmission time overhead with the calculation time overhead, thereby improving the execution efficiency of text training tasks of large language models under limited GPU resources.

[0073] Additionally, in one embodiment, when performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation, which may specifically include:

[0074] If the current text segment has at least two preceding text segments, all preceding text segments except the first text segment will be taken as target preceding text segments in turn; while performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the previous text segment of the target preceding text segment will be loaded; after completing the attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the target preceding text segment will be deleted.

[0075] Among them, if there are multiple previous text segments for the current text segment, the attention calculation can be performed separately based on the key-value pair activation values ​​corresponding to each previous text segment. That is, if there are multiple previous text segments for the current text segment, the current text segment needs to perform multiple attention calculations. In the forward propagation stage, in multiple attention calculations on the same text segment, except for the last attention calculation, the key-value pair activation values ​​of the previous text segment can be loaded at the same time as each attention is executed, so that the loading of the key-value pair activation values ​​is synchronized with the attention calculation.

[0076] For example, if the target text sequence is segmented into 4 text segments. If the current text segment is the 4th text segment, and attention calculation is being performed based on the KV activation value corresponding to the 3rd text segment, the KV activation value corresponding to the 2nd text segment can be loaded from the CPU to the GPU memory first. After completing the attention calculation based on the KV activation value corresponding to the 3rd text segment, the attention calculation is performed based on the KV activation value corresponding to the 2nd text segment, and the KV activation value corresponding to the 2nd text segment in the GPU memory is deleted after this attention calculation is completed. While the attention calculation is being performed based on the KV activation value corresponding to the 2nd text segment, the KV activation value of the 1st text segment is loaded into the GPU memory.

[0077] Therefore, in this embodiment, the attention calculation and the KV activation value transmission are carried out synchronously, thereby reducing the overall time overhead in the forward propagation process and improving the efficiency of the forward propagation.

[0078] Additionally, in one embodiment, when performing model forward propagation on the current text segment, the above-mentioned model training task execution method may further include:

[0079] If the current text segment has a previous text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0080] When executing the model forward propagation for the current text segment, both attention and non-attention computations are included. For example, Embedding, Dropout, LayerNorm, and QKV projection are all non-attention computations. During the non-attention computation phase, activation values ​​other than the KV activation value can be used as target activation values ​​and offloaded to the CPU.

[0081] For example, for the current text segment Chunk i , perform each non-attention calculation in turn, and mark the activation values ​​generated except Ki and Vi as If the current text segment Chunk i If it is not the first text segment Chunk1, then during the non-attention calculation, the target activation value other than the KV activation value corresponding to the previous text segment of the current text segment is deleted. .

[0082] In addition, after executing the attention calculation of a decoder block in the model, the non-attention calculations performed may include: attention calculation output projection, random inactivation (Dropout), ResidualAdd, layer normalization (LayerNorm), feedforward neural network calculation (FFN), Dropout, ResidualAdd, and the LayerNorm and QKV projection of the next decoder block until the next attention calculation is encountered or the model training is completed. i , and all target activation values ​​except KV activation values ​​generated in the current non-attention calculation are collectively marked as If the currently processed text segment is not the first text segment Chunk1, the previous text segment Chunk is unloaded while the current non-attention calculation is in progress. i-1 The corresponding target activation value In addition, for each decoder block in the model, the above attention calculation and non-attention calculation can be repeatedly performed until each decoder block completes the calculation.

[0083] Therefore, in this embodiment, asynchronous offloading of different types of activation values ​​is achieved, and the offloaded data is transferred simultaneously with model computation, thereby masking the time overhead of data transfer. When performing forward propagation of the i-th text segment, the non-KV activation values ​​of the i-1-th text segment are offloaded to CPU memory. When performing backward propagation of the i-th text segment, the i-1-th text segment can be reloaded to the GPU for use in the i-1-th segment's backward propagation. The activation values ​​of the last text segment are not unloaded or loaded.

[0084] During attention calculation, the query vector of the current text segment is sequentially paired with the KV activation values ​​of each preceding text segment to perform partial attention calculation. Simultaneously, the KV activation values ​​generated by the current text segment are asynchronously unloaded. The unloading of non-KV activation values ​​occurs during non-attention calculation time. KV activation unloading / loading is interspersed with the unloading / loading of non-KV activation values, achieving deep overlap between transmission and computation. This scheduling mechanism prevents the high-speed Peripheral Component Interconnect Express (PCIe) bandwidth from being simultaneously occupied by two activation types, thereby improving input / output (I / O) utilization.

[0085] This embodiment implements different scheduling priorities and scheduling timings for KV activation values ​​and non-KV activation values. In view of the higher time sensitivity of KV activation values ​​in cross-segment attention calculations, loading and unloading are performed during attention calculations, and the transmission of KV activation values ​​is considered to be set to high priority, while the transmission of non-KV activation values ​​is set to low priority and performed during non-attention calculations, thereby completing the transmission in stages without blocking the transmission of KV activation values. This hierarchical and refined activation value unloading scheduling strategy of this embodiment can further improve the concurrent efficiency of data transmission and model calculations.

[0086] Additionally, in one embodiment, when performing model back propagation on the current text segment, the above-mentioned model training task execution method may further include:

[0087] If the current text segment has a previous text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

[0088] During back propagation, the current text segment Chunk i , execute the non-attention calculation of the back propagation of the model from back to front until the next attention calculation is required or the back propagation ends. If the text segment currently being processed is not the first text segment Chunk1, the previous text segment Chunk is loaded while the non-attention calculation is being performed. i-1 The target activation value generated during the non-attention calculation in this step is sent to the GPU.

[0089] It is understood that when performing backpropagation attention calculations, the KV activation values ​​of the previous text segments that have not yet participated in the calculation are loaded into the GPU for use in the next attention calculation. For example, while performing attention calculations based on the KV activation values ​​corresponding to the third text segment, the KV activation values ​​corresponding to the second text segment are loaded from the CPU to the GPU.

[0090] In this way, during the backpropagation process, KV activation values ​​and non-KV activation values ​​can be interspersed with unloading and loading, thereby avoiding the simultaneous requisition of PCIe bandwidth by different types of activation values ​​and improving I / O utilization.

[0091] Additionally, in one embodiment, after all text segments of the target text sequence complete backpropagation, the above-mentioned model training task execution method may further include:

[0092] According to the backpropagation results of all text segments of the target text sequence, the model parameters of the trained model and the optimizer state of the model are updated.

[0093] It is understandable that the model back propagation is completed in sequence from back to front for each text segment of the target text sequence until the back propagation is completed. Afterwards, after all text segments have completed the back propagation, the model parameters and optimizer state of the trained model are updated according to the back propagation results stored in the CPU. The model is trained according to the target text sequence in a loop until a preset training termination condition is reached. The preset training termination condition can be that the model output result reaches a preset standard, or reaches a preset number of training times, etc., which is not specifically limited in this embodiment.

[0094] Based on this, this embodiment can break through the graphics memory bottleneck of text sequence training when executing model training tasks, eliminating the need to rely on a large number of GPU resources, reducing the hardware cost and threshold of training. In addition, it can also improve the efficiency of training task execution when GPU resources are limited.

[0095] In this embodiment, a model training task execution method is also provided for a central processing unit. Figure 4 This is a flow chart of the model training task execution method of this embodiment, such as Figure 4 As shown, the following steps are included:

[0096] Step S401, in response to the graphics processor performing the attention calculation in the forward propagation of the model on the current text segment, the graphics processor returns each first reading value to the graphics processor in sequence for the first loading operation of each first reading value; and receives and stores the key-value pair activation value corresponding to the current text segment unloaded by the graphics processor; wherein, each first reading value is the key-value pair activation value corresponding to each predecessor text segment of the current text segment; the first loading operation is initiated by the graphics processor when it is determined that the current text segment has a predecessor text segment; the current text segment is: one of the text segments after the target text sequence is segmented in sequence.

[0097] Step S402: In response to the second loading operation of the second read value when the graphics processor performs the attention calculation of the model back propagation on the current text segment, the second read value is returned to the graphics processor; and in response to the third loading operation of the graphics processor for each first read value in sequence, each first read value is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and the key-value pair activation value corresponding to the current text segment; the gradient value unloaded by the graphics processor is received and stored; the gradient value is obtained by performing the attention calculation based on the second read value and each first read value respectively.

[0098] The above steps S401 to S402 can break through the GPU memory bottleneck of text sequence training and overlap the transmission time overhead with the computation time overhead, thereby improving the execution efficiency of text training tasks for large language models under limited GPU resources. Specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0099] In one embodiment, before the GPU executes the forward propagation of the model, the above-mentioned model training task execution method may further include:

[0100] According to the model initialization parameters corresponding to the model to be trained, the maximum segment size that can be accommodated by the graphics processor is determined; according to the maximum segment size, the target text sequence is segmented to obtain each text segment.

[0101] When segmenting the target text sequence, the model initialization parameters can be used to estimate the maximum segment size that can be accommodated by the GPU memory. , and use this to segment the target text sequence. The corresponding segment size estimation formula can be set according to the model architecture. For example, for the GPT model, the corresponding estimation formula can be:

[0102] ;

[0103] Among them, b, h, L, V, num_param, and gpu_mem are the training batch size, model hidden dimension size, number of model layers, model vocabulary size, model parameter number, and GPU memory capacity respectively.

[0104] In this embodiment, segmented planning can be implemented according to the model characteristics and architecture, thereby further improving the efficiency of model training task execution.

[0105] In one embodiment, the above-mentioned model training task execution method may further include:

[0106] The required memory capacity of the central processing unit is determined based on the model initialization parameters, the target text sequence length, and the maximum segment size. When the required memory capacity is greater than the available memory capacity of the central processing unit, a recalculation strategy is set for the model to be trained.

[0107] The CPU memory size required for training can be estimated based on the model initialization parameters, the target text sequence length, and the maximum segment size that can be accommodated by the GPU memory. For example, for the GPT model, it can be calculated as follows:

[0108] ;

[0109] Among them, s0 represents the maximum sequence length that needs to be trained, that is, the target text sequence length; cpu_mem0 represents the CPU memory size required for training. When the calculated cpu_mem0 exceeds the available memory capacity of the CPU, the recalculation strategy is applied to the activation function and layer normalization module in the training model, and the required CPU memory size and the maximum segment size that can be accommodated by the GPU memory are recalculated. If the required CPU memory size still exceeds the available CPU memory capacity at this time, the recalculation strategy is further applied to the linear layer in the model, and the required CPU memory size and the maximum segment size that can be accommodated by the GPU memory are recalculated again. Recalculation can specifically be a memory optimization technology in deep learning, which reduces the memory usage of the model during training by trading time for space.

[0110] In addition, you can also determine the computation time of each module by performing single-module tests on each module of the model to be trained, and multiply the computation time required for a single text segment by the number of segments to estimate the overall computation time under different recomputation strategies, thereby selecting the recomputation strategy with the shortest computation time required for training.

[0111] In this embodiment, it is considered that after the activation values ​​of each text segment are unloaded to the CPU, the problem of insufficient CPU memory may occur, which will limit the expansion of the length of the trainable text sequence. In this regard, when the CPU memory is insufficient, this embodiment adaptively selects a part of the modules in the model to apply the recalculation strategy based on factors such as the available CPU memory, thereby allowing the training of longer text sequences without increasing the CPU memory. In particular, modules with low computational overhead for recalculation in the model can be preferentially selected to avoid a significant impact of recalculation on the overall efficiency of the training task execution.

[0112] Additionally, this embodiment considers that in some cases, applying recalculation can allow for larger segment sizes, thereby improving GPU computing unit utilization and overcoming the negative impact of recalculation on computational efficiency. Therefore, for different recalculation strategies, not only the capacity of GPU and CPU memory but also computation time are considered. The computation time of each module is obtained through single-module testing, and combined with the activation value usage of each module, the most efficient recalculation strategy and segment size are selected.

[0113] Therefore, this embodiment uses CPU memory as a cache for activation values, splitting and offloading all activation values ​​generated during the process along the sequence dimension. Compared to related technologies, this embodiment can support training longer text sequence lengths in scenarios with sufficient CPU memory resources. Furthermore, a selective recalculation strategy is used to alleviate issues associated with CPU memory limitations, and the strategy is adaptively adjusted through single-module testing to find the optimal balance between computational efficiency and long-sequence scalability.

[0114] It should also be noted that before training, the model structure of the large language model (such as GPT, Llama, GLM) to be trained can be packaged. Specifically, according to the attention calculation modules (Attn i , where i = 1, 2, ..., L, where L is the number of layers of the large language model), and the large language model is divided into L attention calculation blocks (Attn i ), and L+1 non-attention calculation blocks (Not_Attn i ).

[0115] Figure 5 This is a schematic diagram of the structure segmentation of a large language model in this embodiment. Figure 5 As shown, the large language model to be trained can be divided into several attention calculation blocks: attention calculation block 1 (Attn1), attention calculation block 2 (Attn2), and so on; and several non-attention calculation blocks: non-attention calculation block 1 (Not_Attn1), non-attention calculation block 2 (Not_Attn2), non-attention calculation block 3 (Not_Attn3) until non-attention calculation block L+1 (Not_Attn L+1). Among them, attention calculations are performed in each attention calculation block, such as self-attention calculation (selfAttention), and non-attention calculations are performed in each non-attention calculation block, such as embedding calculation (Embedding), QKV projection (OKVproj), linear calculation (Linear, including Linear1 and Linear2), random inactivation calculation (Dropout), layer normalization calculation (LayerNorm), Gaussian error linear calculation (GeLU), loss value calculation (Loss), etc.

[0116] Figure 6 This is a schematic diagram of one of the activation value transmissions in this embodiment. Figure 6 As shown in , the target text sequence is divided into 4 segments, including text segment 1, text segment 2, text segment 3 and text segment 4. In the forward propagation stage, for each layer in the model (such as Figure 6 The three black boxes just below "Text Segment 1" represent layers 1, 2, and 3, respectively. Each layer contains yellow and blue squares. When performing calculations, when performing attention calculations for text segment 1, the KV activation value is transferred (unloading or loading of the KV activation value). Figure 6 When performing the non-attention calculation of text segment 2, the non-KV activation value of text segment 1 is transmitted (in the forward propagation, the non-KV activation value is unloaded, Figure 6 ( The blue squares in the figure represent non-KV activation values.) Similarly, when performing attention calculations on text segment 2, KV activation values ​​are simultaneously transferred. The same applies to text segments 3 and 4. During the backpropagation phase, when performing attention calculations on text segment 4, KV activation values ​​are simultaneously loaded. When performing non-attention calculations on text segment 3, non-KV activation values ​​for text segment 4 are loaded, and so on. During forward and backward propagation, attention calculations are implemented by transferring KV activation values ​​multiple times, and non-KV activation values ​​for the previous text segment are asynchronously offloaded. This achieves a hierarchical and refined offloading scheduling strategy.

[0117] Figure 7 FIG. 1 is a schematic diagram of the local structure of a large language model of this embodiment. Figure 7As shown, the large language model can include L layers, each of which can include an attention module and a feedforward neural network (FFN) module. The attention module includes QKV projection calculation (QKV proj), self-attention calculation (SelfAttention), linear calculation (Linear), and random dropout calculation (Dropout). The FFN module includes linear calculation 1 (Linear1), Gaussian error linear calculation (GeLU), linear calculation 2 (Linear1), and random dropout calculation (Dropout). In addition, layer normalization (LayerNorm) is included between Attention and FFN. The large language model can also include embedding calculation (Embedding) and loss value calculation (Loss).

[0118] Figure 8 This is a schematic diagram of a combination of segmented calculation and activation value transmission in this embodiment, as shown in FIG. Figure 8 As shown in , if the target text sequence is divided into 4 text segments, then according to the segmentation of the target text sequence, text segment 1, text segment 2, text segment 3, and text segment 4 are calculated in sequence. Figure 5During the self-attention calculation process (selfAttention): While performing the attention calculation for text segment 1 (calculating query vector Q1 with the KV activation value KV1 corresponding to the text segment) to obtain the calculation result O_11, the KV activation value KV1 corresponding to text segment 1 is offloaded to the CPU. When performing the attention calculation for text segment 2, an attention calculation is first performed based on the query vector Q2 corresponding to text segment 2 and the KV activation value KV1 corresponding to text segment 1, obtaining the calculation result O_21. Then, a second attention calculation is performed based on the KV activation value KV2 corresponding to text segment 2 and the query vector Q2, obtaining the calculation result O_22. During this second attention calculation, the KV activation value KV2 corresponding to text segment 2 is offloaded to the CPU. When calculating attention for text segment 3, the KV activation value KV1 corresponding to text segment 1, the KV activation value KV2 corresponding to text segment 2, and the KV activation value KV3 corresponding to text segment 3 are sequentially used in conjunction with the query vector Q3 corresponding to the text segment to perform attention calculations, outputting the corresponding calculation results O_31, O_32, and O_33, respectively. When calculating attention for text segment 4, the KV activation value KV1 corresponding to text segment 1, the KV activation value KV2 corresponding to text segment 2, the KV activation value KV3 corresponding to text segment 3, and the KV activation value KV4 corresponding to text segment 4 are sequentially used in conjunction with the query vector Q4 corresponding to the text segment to perform attention calculations, outputting the corresponding calculation results O_41, O_42, O_43, and O_44, respectively. Furthermore, during each attention calculation, the corresponding KV activation value can be unloaded and loaded.

[0119] Additionally, during non-attentional computations (linear computations, layer normalization, activation functions, random dropout, residual connections, etc.), non-KV activation values ​​are offloaded to the CPU. Based on the attentional and non-attentional computations (non-attentional computation result 1, non-attentional computation result 2, non-attentional computation result 3, non-attentional computation result 4), the training outputs (training output 1, training output 2, training output 3, training output 4) corresponding to each text segment are obtained.

[0120] Figure 9 The following is a flowchart of a method for executing a model training task in some embodiments, which is used to train a GPU-1.3B model with a target text sequence of 128K in length (a single sequence has 128×1024 tokens) on a configuration of a single H20 model GPU with a video memory capacity of 96GB and a CPU with a memory capacity of 250GB. Figure 9 As shown, the model training task execution method includes the following steps:

[0121] Step S901, model initialization; wherein, the model can be initialized using the parameters of GPT-1.3B, and Figure 4 The division method shown replaces and packages the GPT-1.3B model structure. Specifically, GPT-1.3B has 24 layers, so the model is divided into 24 segments of attention calculation and 25 segments of non-attention calculation.

[0122] Step S902, determine the segmentation strategy and recalculation strategy for the model; wherein, the GPT-1.3B model and optimizer state require approximately 1.3×16=20.8GB of GPU memory, so there is about 96GB-20.8GB=75.2GB of memory space available for storing segmented activation values. Determine the GPU memory and CPU memory requirements for the activation value, and find that the CPU memory can meet the training requirements. In addition, after applying recalculation to the activation function and layer normalization model, the segment size of the text segment can be increased to 32K, and this setting has the highest overall efficiency under single module testing.

[0123] Step S903 , segmenting the target text sequence; the input target text sequence is 128K in length, and the target text sequence is divided into 4 segments, each of which is 32K in length.

[0124] In step S904, during the forward propagation phase, each text segment is processed sequentially while asynchronously offloading the non-KV activation values ​​of the previous segment to the CPU. The specific transmission of KV activation values ​​and non-KV activation values, as well as the execution mechanisms of attention and non-attention calculations, can be found in the above embodiments and will not be further described.

[0125] In step S905, during the backpropagation phase, each text segment is processed sequentially while asynchronously loading non-KV activation values ​​from the CPU to the GPU. The attention calculation process fully handles the unloading and loading of KV activation values, but does not handle the transmission of non-KV activation values. For details, please refer to the above embodiment and will not be repeated here.

[0126] Step S906, repeat S904 and S905 until the model converges or reaches a predetermined training round, and then end the training.

[0127] The above steps S901 to S906 break through the graphics memory bottleneck of text sequence training by utilizing the asynchronous transmission mechanism of segmented calculation of text and activation values. Compared with the distributed training method of existing related technologies, this embodiment does not need to rely on more GPU resources, but adopts relatively low-cost CPU memory resources, which reduces the training cost and threshold. Through the scheduling mechanism of asynchronous transmission of activation values, the time overhead of transmission and the time overhead of calculation are overlapped. Compared with related technologies, additional computing overhead is avoided and the overall efficiency is higher. In addition, by using CPU memory as a cache space for activation values, all activation values ​​generated during the training process are segmented and unloaded in the sequence dimension. In the scenario where CPU memory resources are sufficient, it can support training of longer text sequence lengths compared to existing related technologies.

[0128] This embodiment also provides a graphics processor for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. The terms "module," "unit," "subunit," etc. used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0129] Figure 10 is a structural block diagram of the graphics processor 10 of this embodiment, as shown in FIG. Figure 10 As shown, the graphics processor 10 includes: an acquisition module 12, a forward propagation module 14 and a back propagation module 16;

[0130] An acquisition module 12 is used to obtain the current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence; a forward propagation module 14 is used to, when executing model forward propagation on the current text segment, if the current text segment has a predecessor text segment, then when executing attention calculation, load the key-value pair activation values ​​corresponding to each predecessor text segment from the central processing unit in sequence to perform attention calculation; at the same time, unload the key-value pair activation values ​​corresponding to the current text segment to the central processing unit; a backward propagation module 16 is used to, when executing model backward propagation on the current text segment, if the current text segment has a predecessor text segment, then when executing attention calculation, load the forward propagation output result and key-value pair activation value corresponding to the current text segment from the central processing unit, and load the attention activation value corresponding to each predecessor text segment in sequence, perform attention calculation with the forward propagation output result and key-value pair activation value corresponding to the current text segment, and obtain a gradient value; unload the gradient value to the central processing unit.

[0131] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0132] The above-mentioned graphics processor 10 can break through the GPU memory bottleneck of text sequence training and overlap the transmission time overhead with the calculation time overhead, thereby improving the execution efficiency of text training tasks of large language models under limited GPU resources.

[0133] In one embodiment, the forward propagation module 14 is specifically used to, if the current text segment has at least two preceding text segments, take all preceding text segments except the first text segment as target preceding text segments in turn; while performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, load the key-value pair activation value corresponding to the previous text segment of the target preceding text segment; after completing the attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, delete the key-value pair activation value corresponding to the target preceding text segment.

[0134] In one embodiment, the forward propagation module 14 is also used to unload the target activation value corresponding to the previous text segment of the current text segment to the central processing unit when performing non-attention calculation on the current text segment if there is a previous text segment of the current text segment; the target activation value is an activation value other than the key-value pair activation value.

[0135] In one embodiment, the back propagation module 16 is also used to load the target activation value corresponding to the previous text segment of the current text segment from the central processing unit when performing non-attention calculation on the current text segment if there is a previous text segment of the current text segment; the target activation value is an activation value other than the key-value pair activation value.

[0136] In one embodiment, the graphics processor 10 further includes an update module, which is used to update the model parameters of the trained model and the optimizer state of the model according to the back propagation results of all text segments of the target text sequence.

[0137] This embodiment also provides a model training system. Figure 11 This is a schematic diagram of the structure of the model training system 11 of this embodiment. Figure 11 As shown, the model training system 11 includes a graphics processor 112 and a central processing unit 114; wherein: the graphics processor 112 is used to execute the model training task execution method for the graphics processor provided in any of the above embodiments; the central processing unit 114 is used to execute the model training task execution method for the central processing unit provided in any of the above embodiments.

[0138] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0139] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0141] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0142] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0143] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for executing a model training task, characterized in that: For a graphics processor, the method includes: Obtaining a current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence; When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each of the preceding text segments are sequentially loaded from the central processing unit for attention calculation; at the same time, the key-value pair activation values ​​corresponding to the current text segment are unloaded to the central processing unit; When performing model back propagation on the current text segment, if there is a preceding text segment for the current text segment, then when performing attention calculation, the forward propagation output result and key-value pair activation value corresponding to the current text segment are loaded from the central processing unit, and the attention activation value corresponding to each of the preceding text segments is loaded in turn, and attention calculation is performed on the forward propagation output result and key-value pair activation value corresponding to the current text segment to obtain a gradient value; and the gradient value is unloaded to the central processing unit.

2. The model training task execution method according to claim 1, characterized in that: When performing model forward propagation on the current text segment, if the current text segment has a preceding text segment, then when performing attention calculation, the key-value pair activation values ​​corresponding to each preceding text segment are sequentially loaded from the central processing unit for attention calculation, including: If the current text segment has at least two preceding text segments, all preceding text segments except the first text segment are sequentially used as target preceding text segments; While performing attention calculation based on the key-value pair activation value corresponding to the target preceding text segment, loading the key-value pair activation value corresponding to the previous text segment of the target preceding text segment; After completing the attention calculation according to the key-value pair activation value corresponding to the target preceding text segment, the key-value pair activation value corresponding to the target preceding text segment is deleted.

3. The model training task execution method according to claim 1, characterized in that: When performing model forward propagation on the current text segment, the method further includes: If the current text segment has a preceding text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is unloaded to the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

4. The model training task execution method according to claim 1, characterized in that: When performing model back propagation on the current text segment, the method further includes: If the current text segment has a preceding text segment, then when performing non-attention calculation on the current text segment, the target activation value corresponding to the previous text segment of the current text segment is loaded from the central processing unit; the target activation value is an activation value other than the key-value pair activation value.

5. The method for executing a model training task according to any one of claims 1 to 4, characterized in that: After all text segments of the target text sequence complete back propagation, the method further includes: Based on the back-propagation results of all text segments of the target text sequence, the model parameters of the trained model and the optimizer state of the model are updated.

6. A method for executing a model training task, characterized in that: For a central processing unit, the method includes: In response to the graphics processor performing the attention calculation in the forward propagation of the model on the current text segment, the graphics processor sequentially performs the first loading operation on each first read value, and returns each first read value to the graphics processor respectively; and receives and stores the key-value pair activation value corresponding to the current text segment unloaded by the graphics processor; wherein each first read value is a key-value pair activation value corresponding to each preceding text segment of the current text segment; the first loading operation is initiated by the graphics processor when it is determined that the current text segment has a preceding text segment; the current text segment is: one of the text segments after the target text sequence is sequentially segmented; In response to the second loading operation for the second read value when the graphics processor performs the attention calculation of the model back propagation on the current text segment, the second read value is returned to the graphics processor; and in response to the third loading operation of the graphics processor for each first read value in sequence, each first read value is returned to the graphics processor respectively; wherein the second read value is the forward propagation output result and key-value pair activation value corresponding to the current text segment; receiving and storing the gradient value unloaded by the graphics processor; the gradient value is obtained by performing the attention calculation based on the second read value and each first read value respectively.

7. The model training task execution method according to claim 6, characterized in that: Before the graphics processor performs forward propagation of the model, the method further includes: Determining a maximum segment size that can be accommodated by the graphics processor based on model initialization parameters corresponding to the model to be trained; According to the maximum segment size, the target text sequence is segmented to obtain text segments.

8. The model training task execution method according to claim 7, characterized in that: The method further comprises: Determining the required memory capacity of the central processing unit according to the model initialization parameters, the target text sequence length and the maximum segment size; When the required memory capacity is greater than the available memory capacity of the central processing unit, a recalculation strategy is set for the model to be trained.

9. A graphics processor, characterized in that: include: Acquisition module, forward propagation module and back propagation module; The acquisition module is used to acquire a current text segment; wherein the current text segment is: one of the text segments after the target text sequence is segmented in sequence; The forward propagation module is configured to, when performing model forward propagation on the current text segment, if there is a previous text segment for the current text segment, sequentially load the key-value pair activation values ​​corresponding to each of the previous text segments from the central processing unit to perform attention calculation; and simultaneously, unload the key-value pair activation values ​​corresponding to the current text segment to the central processing unit; The back propagation module is used to, when performing model back propagation on the current text segment, if the current text segment has a preceding text segment, load the forward propagation output result and key-value pair activation value corresponding to the current text segment from the central processing unit when performing attention calculation, and sequentially load the attention activation value corresponding to each of the preceding text segments, perform attention calculation with the forward propagation output result and key-value pair activation value corresponding to the current text segment, and obtain a gradient value; and unload the gradient value to the central processing unit.

10. A model training system, characterized in that: include: Graphics processor and CPU; The graphics processor is used to execute the model training task execution method according to any one of claims 1 to 5; The central processing unit is used to execute the model training task execution method described in any one of claims 6 to 8.

Images