Model training method and device based on fine-grained scheduling, equipment and storage medium

Abstract

The invention relates to a model training method and device based on fine-grained scheduling, equipment and a storage medium. The method comprises the steps of obtaining a first sample sequence; inputting the first sample sequences into a model for training, allocating video memory spaces for the first sample sequences in the training process, and performing operation on the first sample sequences based on independent threads; under the condition that any first sample sequence completes back propagation, releasing the video memory space occupied by the first sample sequence, and inputting a new second sample sequence into the model for training; the problem that operation resources are wasted in the training process of a large language model is solved.

Description

Technical Field

[0001] The present application relates to the field of model training, and in particular to a model training method, apparatus, device and storage medium based on fine-grained scheduling. Background Art

[0002] At present, mainstream generative large language models, such as GPT and LLaMA, all use decoder-only transformer architecture. The main part of this model architecture is multiple decoder blocks. Each decoder block includes a multi-head attention layer, a fully connected layer, and a residual connection layer. During the model training phase, the multi-head attention layer calculates each token in the sample sequence separately from each token in the sample sequence that is further ahead. Therefore, the time complexity and memory complexity of traditional attention calculation are both o(N 2 ) level (N is the sequence length), which is significantly higher than the o(N) level of other parts. Today's large language model training and inference usually use flash attention technology. This technology uses a block calculation strategy and an online-softmax algorithm to optimize the forward and backward propagation of the attention operator. During forward propagation, each word is no longer calculated simultaneously with the previous word in the form of matrix multiplication, but is calculated with each previous word block in turn, thereby reducing the memory complexity of the attention calculation forward and backward propagation to the o(N) level, while the time complexity remains unchanged at o(N). 2 ). During back propagation, the block calculation strategy is also used to replace the matrix multiplication in the original method, so the back propagation time complexity is also o(N 2 This technology improves the speed of attention operations and the throughput of large language model training and inference processes by reducing GPU (Graphics Processing Unit) memory consumption and GPU SRAM (Static Random-Access Memory) IO (Input / Output) overhead.

[0003] On the other hand, in the training of large language models, in order to fully utilize the parallelism of GPU computing, it is usually necessary to group multiple samples into a batch and input them into the model. Since real training data sets usually consist of a large number of short texts and a small number of long texts, there is a problem of uneven lengths of samples in the same batch. To address this, the previous approach was to use a special Pad Token to pad each sample in the batch to the same length. This approach is more convenient to implement. When calculating attention, it can be directly divided into individual samples according to the batch size (number of iterated samples) dimension and processed separately. However, if Figure 1 As shown in the figure, the introduced Pad Token will participate in various calculations like other tokens and generate activation value memory occupation, resulting in a waste of computing resources and memory space.

[0004] At present, there is a problem of waste of computing resources during the training process of large language models in related technologies, and no effective solution has been proposed yet. Summary of the invention

[0005] Based on this, the present application provides a model training method, apparatus, device and storage medium based on fine-grained scheduling to solve the problem of waste of computing resources during the training process of large language models.

[0006] In a first aspect, the present application provides a model training method based on fine-grained scheduling, comprising:

[0007] Acquire a first sample sequence;

[0008] Inputting the first sample sequences into the model for training, allocating video memory space to each of the first sample sequences during the training process, and performing operations on each of the first sample sequences based on a separate thread;

[0009] When any first sample sequence completes back propagation, the video memory space occupied by the first sample sequence is released, and a new second sample sequence is input into the model for training.

[0010] In one embodiment, allocating video memory space to each of the first sample sequences during training includes:

[0011] Determine a plurality of video memory blocks according to the size of the video memory space, and configure a physical address for each of the video memory blocks; wherein each of the video memory blocks can accommodate activation values ​​generated by a preset number of word units;

[0012] According to the length of the first sample sequence and the preset number, a virtual block table is constructed in the video memory space for each of the first sample sequences, and physical addresses corresponding to a plurality of the video memory blocks are stored in the virtual block table.

[0013] In one embodiment, when any first sample sequence completes back propagation, releasing the video memory space occupied by the first sample sequence includes:

[0014] In the case where any first sample sequence completes back propagation, determining a virtual block table of the first sample sequence;

[0015] According to the physical address stored in the virtual block table, the corresponding video memory space is released.

[0016] In one of the embodiments, when any first sample sequence completes back propagation, the method further includes:

[0017] The gradient values ​​of each of the first sample sequences obtained during the back propagation are obtained, and the gradient values ​​are accumulated to obtain a gradient accumulation value.

[0018] In one of the embodiments, when any first sample sequence completes back propagation, the method further includes:

[0019] Determine whether the number of the first sample sequence that completes the back propagation reaches a preset threshold;

[0020] If the number of the first sample sequences that complete back propagation reaches the preset threshold, updating the first model parameters based on the gradient accumulation value, and performing training based on the first model parameters after inputting the second sample sequence into the model;

[0021] If the number of the first sample sequences that have completed back propagation does not reach the preset threshold, after the second sample sequence is input into the model, the second model parameters are continuously used for training.

[0022] In one embodiment, determining whether the number of first sample sequences that complete back propagation reaches a preset threshold; if the number of first sample sequences that complete back propagation reaches the preset threshold, updating the first model parameter based on the gradient accumulation value; if the number of first sample sequences that complete back propagation does not reach the preset threshold, continuing to use the second model parameter for training, includes:

[0023] Determine whether the value of the counter reaches the preset threshold; wherein, whenever any first sample sequence completes reverse propagation, the value of the counter is accumulated;

[0024] If the counter reaches the preset threshold, determining the first model parameter according to a preset flag bit, and after updating the first model parameter, inverting the flag bit;

[0025] If the counter does not reach the preset threshold, the second model parameter is determined according to the preset flag bit.

[0026] In one embodiment, obtaining a first sample sequence includes:

[0027] Determine the number of words that can be accommodated in the current video memory space;

[0028] Determine the occupied space size of the activation value of the sample sequence according to the number of word units;

[0029] The number of the first sample sequences to be input into the model is determined according to the size of the space occupied by the activation value.

[0030] In a second aspect, the present application also provides a model training device based on fine-grained scheduling, comprising:

[0031] An acquisition module, used for acquiring a first sample sequence;

[0032] An input module, used for inputting the first sample sequences into a model for training, allocating video memory space to each of the first sample sequences during the training process, and performing operations on each of the first sample sequences based on a separate thread;

[0033] The scheduling module is used to release the video memory space occupied by any first sample sequence when the back propagation of the first sample sequence is completed, and input the new second sample sequence into the model for training.

[0034] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0036] In the above-mentioned model training method, apparatus, device and storage medium based on fine-grained scheduling, multiple sample sequences no longer always perform operations on each neural network layer in the video memory space synchronously, but instead put each sample sequence into multiple separate threads, allowing them to asynchronously complete the operations of each layer of the model at their own pace. Whenever a sample sequence ends back propagation, the video memory space it occupies is released, and a new sample sequence is input to achieve scheduling with a single sample sequence as the granularity. As a result, idle computing resources can be used in a timely manner to solve the problem of waste of computing resources in the training process of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an illustration of the principle of large language model training in related technologies. Figure 1 ;

[0038] Figure 2 A hardware structure block diagram of a terminal of a model training method based on fine-grained scheduling in one embodiment;

[0039] Figure 3 This is an illustration of the principle of large language model training in related technologies. Figure 2 ;

[0040] Figure 4 This is an illustration of the principle of large language model training in related technologies. Figure 3 ;

[0041] Figure 5 is a flowchart of a model training method based on fine-grained scheduling in one embodiment;

[0042] Figure 6 A schematic diagram of the principle of model training based on fine-grained scheduling in one embodiment;

[0043] Figure 7 A schematic diagram of storing a sample sequence in a video memory space in one embodiment;

[0044] Figure 8 is a flowchart of a video memory management method in one embodiment;

[0045] Fig. 9 A schematic diagram of the principle of video memory management in one embodiment;

[0046] Fig.10 is a flow chart of a gradient updating method in one embodiment;

[0047] Fig.11 is a flowchart of a model training method based on fine-grained scheduling in another embodiment;

[0048] Fig.12 The structure block diagram of a model training device based on fine-grained scheduling in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] In one embodiment, a computer device is provided, and the method embodiment provided in the present application can be executed in the computer device. The computer device can be a terminal, a computer, or a similar computing device, such as running on a terminal, Figure 2 FIG. 1 is a hardware structure block diagram of a terminal of a model training method based on fine-grained scheduling according to an embodiment of the present application. Figure 2 As shown, the terminal may include one or more ( Figure 2 Only one is shown in the figure) processor 101 and memory 102 for storing data, wherein processor 101 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 103 and an input and output device 104 for communication functions. It can be understood by those skilled in the art that Figure 2 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown.

[0052] The memory 102 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model training method based on fine-grained scheduling in this embodiment. The processor 101 executes various functional applications and data processing by running the computer program stored in the memory 102, that is, to implement the above method. The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 102 may further include a memory remotely arranged relative to the processor 101, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0053] The transmission device 103 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 103 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 103 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0054] In view of the problems mentioned in the background technology, an improved method is as follows Figure 3 As shown in the figure, the shorter sequences in the data set are spliced ​​into longer sequences through a heuristic algorithm, which greatly reduces the number of pad tokens and reduces the waste of video memory space. However, the short sequences spliced ​​into a long sequence will be regarded as the same sequence in the attention layer, and cross operations will still occur between their words. Although these operation results will be shielded by the modified attention mask and will not affect the model training, invalid operations are added. Another improvement method is to connect all sequences end to end to form a single long sequence for processing. Although this approach avoids the waste of video memory space and invalid calculations, it will introduce the problem of uneven load due to the different lengths of different samples.

[0055] Specifically, in Pytorch, for each neural network layer of the model (such as linear layer, attention layer, residual connection layer, etc.), all samples need to complete the calculation of the layer before returning the result and entering the subsequent layer together. Since the time complexity of the forward propagation and back propagation of the attention operator is o(N 2), uneven sample lengths will result in different computation times for each sample. In addition, since each sample is independent in the attention operation, different samples need to be assigned to different SMs (stream multiprocessors) for processing. Once a sequence is assigned to an SM, all operations can only be completed on that SM and cannot be reallocated to other SMs for operation. Therefore, in an attention calculation, if an SM is assigned to some shorter sequences, it will end the operation earlier and enter an idle state, waiting for other SMs to complete the operation of the remaining samples, which causes the idling problem in the attention operation. Figure 4 As shown in the figure, since each decoder block of the model contains an attention layer, this waiting will occur in the forward propagation and back propagation of each layer, which causes a waste of computing resources.

[0056] Based on the analysis of the above situation, this embodiment provides a model training method based on fine-grained scheduling to solve the problem of waste of computing resources during the training process of a large language model. Figure 5 The flowchart of the model training method based on fine-grained scheduling is shown in Figure 2. Figure 2 The terminal in is used as an example to illustrate, including the following steps:

[0057] Step S101, obtaining a first sample sequence.

[0058] A large language model is a machine learning model with a large number of parameters that needs to be trained on a large amount of text data to understand and generate natural language, so that it can perform various natural language processing tasks such as text generation, translation, question answering, etc. A sample sequence is a series of inputs or outputs used to train or evaluate a model. In the context of natural language processing, a sample sequence can be a sequence of text or a group of words. For example, when training a language model, a sample sequence can be a sentence consisting of several consecutive words, and the goal of the model is to predict what the next possible word is.

[0059] In this step, the number of tokens that can be accommodated by the current video memory space can be determined; the size of the space occupied by the activation value of the sample sequence can be determined based on the number of tokens; and the number of the first sample sequence to be input into the model can be determined based on the size of the space occupied by the activation value. For example, in the data preprocessing and training initialization stage, the number of tokens that can be accommodated by the video memory space is calculated based on the size of the video memory space available to the model and the model scale. Since the size of the space occupied by the activation value is proportional to the number of tokens in the sample sequence, and the model structure will not change during training, the result can be calculated more accurately. An initial batch is randomly selected from the pre-tokenized sample sequence so that the total number of tokens in the initial batch does not exceed the number of tokens that can be accommodated. It should be noted that the first sample sequence can be a sample sequence of an initial batch or a sample sequence of an intermediate batch, and this application does not limit this.

[0060] Step S102 : input the first sample sequences into the model for training, allocate video memory space to each first sample sequence during the training process, and perform calculations on each first sample sequence based on a separate thread.

[0061] In this step, the first sample sequence of the current batch can be input into the model for forward propagation and back propagation operations. In this process, each first sample sequence is processed using a separate thread, and an independent kernel function (CUDA kernel) is always used for operation in the video memory space. Among them, the grid size of the kernel function is set to 1, so that each sample sequence obtained after each sample sequence is split by the attention head is assigned to a separate stream multiprocessor for execution. The model consists of a multi-layer neural network, such as a linear layer, an attention layer, or a residual connection layer. When a sample sequence completes the operation of any neural network layer, the sample sequence does not wait for other sample sequences to complete, but directly enters the operation of the subsequent neural network layer.

[0062] Step S103, when any first sample sequence completes back propagation, the video memory space occupied by the first sample sequence is released, and a new second sample sequence is input into the model for training.

[0063] In this step, when a sample sequence completes back propagation, the video memory space it occupies is released, and the number of tokens that can be accommodated in the video memory space is estimated based on the free video memory size and the model scale. If the free video memory is sufficient, a certain number of new sample sequences (i.e., the second sample sequence) are randomly selected from the data set and added to the batch, and video memory is allocated for their activation values, and then the process returns to step S102 to continue execution.

[0064] Specifically, Figure 6As shown in the figure, during the forward propagation process, when a sample sequence completes the calculation of the current layer, it directly enters the next layer of calculation; when the forward propagation ends, it will also directly enter the calculation of the loss function, and then directly enter the back propagation process. During the back propagation process, each time the calculation of a decoding module (decoder block) is completed, it will directly enter the calculation of the previous decoding module. When a sample sequence finishes back propagation, the video memory space it occupies is released, and it is checked whether the free video memory is sufficient. If it is sufficient, one or more new sample sequences are added to the batch for calculation. In this way, the idle stream multiprocessors can be used in a timely manner to avoid the waste of computing resources caused by the waiting of the stream multiprocessors, thereby solving the problem of low utilization of video memory and computing resources caused by the idleness of the stream multiprocessors due to uneven load.

[0065] It should be noted that in this application, the first sample sequence and the second sample sequence are distinguished in terms of the time sequence of their input into the model, and do not represent a distinction between their own characteristics. In some embodiments, the first sample sequence is called the old sample sequence, and the second sample sequence is called the new sample sequence. GPU, video memory, and video memory space have the same meaning and can be interchangeable.

[0066] In the above steps S101 to S103, multiple sample sequences no longer always perform operations on each neural network layer synchronously in the video memory space. Instead, each sample sequence is placed in multiple separate threads, allowing them to asynchronously complete the operations of each layer of the model at their own pace. Whenever a sample sequence finishes backpropagation, the video memory space it occupies is released, and a new sample sequence is input to achieve scheduling with a single sample sequence as the granularity. In this way, idle computing resources can be used in a timely manner to solve the problem of waste of computing resources in the training process of large language models.

[0067] In the above embodiments, the asynchronous operation of each sample sequence brings about problems in video memory management and gradient update. The following embodiments will provide corresponding solutions to these two problems.

[0068] In terms of video memory management, before inputting a new sample sequence into the model, it is necessary to determine whether the current video memory space can accommodate the activation values ​​required by the new sample sequence. If it cannot accommodate, the input of the new sample sequence needs to be suspended. In addition, since a sample sequence may complete all calculations and exit at any time, or a new sample sequence may be input into the model to start calculations, it is necessary to release the video memory space occupied by the activation values ​​of the old sample sequence, or allocate new video memory space for the activation values ​​of the new sample sequence. However, the activation value space required by the sample sequence is proportional to the length of the sequence, and the GPU video memory lacks a paging virtual memory management mechanism similar to the CPU memory. Each allocated video memory space is continuous physical video memory. Therefore, if Figure 7As shown in Figure 1, when a shorter sequence completes the operation before other sequences, the released video memory space may not be enough to store the activation value of the new long sequence. It can be seen that the imbalance of sample sequence length will cause external fragmentation in the video memory, resulting in reduced video memory utilization.

[0069] To solve the above problem, in one embodiment, the above step S102, allocating video memory space to each first sample sequence during the training process, can be implemented by the following method:

[0070] Determine multiple video memory blocks according to the size of the video memory space, and configure a physical address for each video memory block; wherein each video memory block can accommodate activation values ​​generated by a preset number of word units; construct a virtual block table in the video memory space for each first sample sequence according to the length and preset number of the first sample sequence, and store the physical addresses corresponding to the multiple video memory blocks in the virtual block table. Further, when any first sample sequence completes back propagation, determine the virtual block table of the first sample sequence; release the corresponding video memory space according to the physical address stored in the virtual block table. After the release, obtain a new second sample sequence and input it into the model for training.

[0071] In this embodiment, in order to solve the problems caused by these external fragments, the GPU memory is divided into multiple memory blocks of fixed size, and the activation value of each sample is stored in multiple discrete memory blocks. At the same time, a memory block table is constructed for each sample to record the physical addresses of each memory block occupied by its activation value, so as to achieve efficient management of the memory.

[0072] For example, Figure 8 A flowchart of the video memory management method of this embodiment is provided. Figure 8 As shown, the process includes the following steps:

[0073] Step S201, as Fig. 9 As shown, before starting training, the video memory space is logically divided into multiple video memory blocks, and the size of each video memory block is set to S. A virtual block table is constructed in the video memory space for each sample sequence, and the size of each virtual block table is max_length / S. Among them, max_length is the upper limit of the length of the sample sequence in the data set, and this value can be determined before the start of training. A video memory block can store the activation values ​​generated by S words in the sample sequence. Since the storage occupancy of the virtual block table is smaller than the activation value, there is no need to worry about memory fragmentation. The physical addresses of each video memory block occupied by the activation value of the sample sequence can be stored in the virtual block table.

[0074] Step S202: When a sample sequence completes the back propagation, the video memory space corresponding to the sample sequence in the virtual block table is released.

[0075] Step S203, calculating the total number of words that can be currently accommodated according to the total amount of free video memory in the video memory space.

[0076] Step S204, based on the result obtained in the previous step, randomly select new sample sequences to be input into the model from the data set, and record the total number of word units in the selected sample sequences, until the nth sample sequence is selected, and the number of word units exceeds the total number of word units that can be accommodated in the video memory space calculated in the previous step.

[0077] Step S205: if n≥2, input the first n-1 sample sequences into the model, and allocate seq_length / S video memory blocks (rounded up) to each sample sequence, and record the physical address of each video memory block in the virtual block table in sequence, where seq_length is the length of the sample sequence; continue to run the current step until the next sample sequence completes back propagation, and then return to step S202.

[0078] Step S206: if n is 1, no sample sequence is input until the next sample sequence completes back propagation, and then returns to step S202.

[0079] In terms of gradient updates, since each sample sequence is operated asynchronously, there will be a sample sequence in the process of forward propagation at any time. However, each layer of a sample sequence in the forward propagation must use the same model parameters. If the model parameters are updated directly in the middle, it will affect the gradient obtained by the back propagation of this sample sequence, thereby affecting the overall training effect.

[0080] To solve the above problem, in one embodiment, Fig.10 A flowchart of a gradient updating method is provided, such as Fig.10 As shown, the process includes the following steps:

[0081] Step S301, when any first sample sequence completes back propagation, obtain the gradient value obtained by each first sample sequence in the process of back propagation, and accumulate each gradient value to obtain a gradient accumulation value.

[0082] Step S302, determine whether the number of first sample sequences that have completed back propagation has reached a preset threshold; if the number of first sample sequences that have completed back propagation has reached the preset threshold, execute step S303; otherwise, execute step S304. In this step, a counter can be used to count the sample sequences that have completed the operation. Whenever any first sample sequence completes back propagation, the counter is accumulated (for example, the value of the counter is increased by 1), so as to use the counter to determine whether the preset threshold has been reached.

[0083] Step S303, updating the first model parameters based on the gradient accumulation value, and training based on the first model parameters after inputting the second sample sequence into the model. In this step, if the counter reaches a preset threshold, the first model parameters can be determined according to a preset flag bit, and after updating the first model parameters, the flag bit is inverted.

[0084] Step S304, after the second sample sequence is input into the model, the second model parameters are continuously used for training. In this step, if the counter does not reach the preset threshold, the second model parameters are determined according to the preset flag bit.

[0085] In this embodiment, in order to solve the problem of gradient update, two copies of model parameters are saved in the model at the same time, and an additional video memory with the same space as the model parameters is allocated to accumulate the gradients obtained by the back propagation of each sample sequence; the two copies of model parameters are used in turn for the forward propagation of new sample sequences. Each time the model parameters need to be updated, only one copy of the model parameters that are not used by the sample sequence is updated. After the update, the sample sequence that is already in the forward propagation process still uses the model parameters that are currently being used, while the sample sequence that has just entered the model uses the other updated model parameters.

[0086] Furthermore, in one embodiment, a positive integer B and a counter can be pre-set during the data preprocessing and training initialization phase. Whenever a sample sequence completes back propagation, the gradient obtained in the back propagation is accumulated, and the counter is incremented by 1. Whenever the counter reaches the pre-set B value, the unused model parameters are updated once using the accumulated gradient value obtained by back propagating the B sample sequences, and the counter is cleared. If the counter does not reach the positive integer B, when a new sample sequence is input into the model, the currently used model parameters are still used. Specifically, Fig.11 A flowchart of the model training method based on fine-grained scheduling of this embodiment is provided, such as Fig.11 As shown, the process includes the following steps:

[0087] Step S401: When the model starts training, the average memory space required for each token and the configuration parameters are calculated based on the GPU memory and model architecture. Specifically, it includes:

[0088] Assuming that there is 16GB of free video memory in the GPU, according to the model architecture, it is calculated that each token needs to occupy 8MB of space for activation value on average, so the GPU can store the activation values ​​of 2000 tokens. Sample sequences are randomly selected from the data set, and the number of tokens in the sample sequences is 50, 100, 200, 700, 250, and 800 respectively. The total number of tokens in the six sample sequences is 2100, which is more than 2000. Then the first five sample sequences (total number of tokens is 1300) are input into the model.

[0089] Two identical copies of the model parameters are stored in the video memory space, namely model parameters No. 1 and model parameters No. 2. At the same time, a space equal to the model parameter video memory is allocated to accumulate the gradients returned by the sample sequence. A counter is set to be initially 0, the threshold is preset to be a positive integer B=16, and a flag F is set to be initially 0 to indicate which version of the model parameters is used for the sample sequence newly entering the model.

[0090] Step S402, constructing virtual block tables and allocating video memory space for the multiple sample sequences added to the batch, at which time the flag F is 0, and each of the multiple sample sequences uses model parameter No. 1 for forward propagation and backward propagation in the model.

[0091] Step S403: Whenever a sample sequence completes back propagation, the gradient value is accumulated, the counter is incremented by 1, the video memory space of the sample sequence is released, the new sample sequence is added to the batch, and a virtual block table is constructed and video memory space is allocated for the sample sequence newly added to the batch. For example, if the first and second sample sequences complete back propagation in sequence, the additional token capacity that can be accommodated in the video memory space is 850, and the sixth sample sequence can be considered to be added to the batch. Continue to randomly select the seventh and eighth sample sequences from the data set in turn, and the sample sequence lengths are 30 and 50 respectively, and then add the seventh sample sequence to the batch.

[0092] Step S404, determine whether the counter reaches a positive integer B; if the counter reaches a positive integer B, execute step S405; otherwise, execute step S409.

[0093] Step S405, determine the flag bit F; if the flag bit F is 0, execute step S406; if the flag bit F is 1, execute step S407.

[0094] In step S406, the flag bit F is 0, and the model parameter No. 2 is updated based on the gradient accumulation value. Then the flag bit F is negated and step S408 is executed.

[0095] In step S407, the flag bit F is 1, and the model parameter No. 1 is updated based on the gradient accumulation value. Then the flag bit F is inverted and step S408 is executed.

[0096] In step S408, the counter and the gradient accumulation value are set to 0, and the process returns to step S403. At this time, the sample sequence still being calculated in the video memory space still uses the previously used model parameters.

[0097] Step S409, determine the flag bit; if the flag bit F is 0, execute step S410; if the flag bit F is 1, execute step S411.

[0098] In step S410, the flag F is 0. When a new sample sequence is input into the model, it is made to use model parameter No. 1 for forward propagation and backward propagation, and the process returns to step S403.

[0099] In step S411, the flag F is 1. When a new sample sequence is input into the model, the model parameter No. 2 is used for forward propagation and backward propagation, and the process returns to step S403.

[0100] Repeat the above steps until the model training is completed.

[0101] In this embodiment, through fine-grained sample scheduling, each sample sequence in a batch is allowed to asynchronously perform forward / backward propagation in the model, which solves the problem of waste of computing resources caused by unbalanced sequence length and uneven computational overhead of different sequences during large model training. By setting two sets of model parameters and using the two sets of model parameters alternately for forward / backward propagation, the training results are ensured to be correct and reliable.

[0102] In one embodiment, a model training device based on fine-grained scheduling is provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below may implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0103] Fig.12 is a structural block diagram of the model training device based on fine-grained scheduling in this embodiment, such as Fig.12 As shown, the device comprises:

[0104] An acquisition module, used for acquiring a first sample sequence;

[0105] An input module, used for inputting the first sample sequence into the model for training, allocating video memory space to each first sample sequence during the training process, and performing operations on each first sample sequence based on a separate thread;

[0106] The scheduling module is used to release the video memory space occupied by any first sample sequence when the back propagation of the first sample sequence is completed, and input the new second sample sequence into the model for training.

[0107] The implementation solution for solving the problem provided by the device is similar to the implementation solution recorded in the above method. Therefore, the specific limitations in one or more embodiments of the model training device based on fine-grained scheduling provided below can be found in the above limitations on the model training method based on fine-grained scheduling, and will not be repeated here.

[0108] In one embodiment, the input module includes a video memory management module, which is used to determine multiple video memory blocks according to the size of the video memory space and configure a physical address for each video memory block; wherein each video memory block can accommodate activation values ​​generated by a preset number of word units; according to the length and preset number of the first sample sequence, a virtual block table is constructed in the video memory space for each first sample sequence, and the physical addresses corresponding to the multiple video memory blocks are stored in the virtual block table.

[0109] In one embodiment, the scheduling module is used to determine the virtual block table of any first sample sequence when the back propagation of the first sample sequence is completed; and release the corresponding video memory space according to the physical address stored in the virtual block table.

[0110] In one embodiment, the device further includes: a gradient updating module, configured to obtain the gradient values ​​obtained by each first sample sequence during the back propagation process, and accumulate the gradient values ​​to obtain a gradient accumulation value.

[0111] In one embodiment, the gradient update module is further used to determine whether the number of the first sample sequences that have completed back propagation reaches a preset threshold; if the number of the first sample sequences that have completed back propagation reaches the preset threshold, the first model parameters are updated based on the gradient accumulation value, and after the second sample sequence is input into the model, training is performed based on the first model parameters.

[0112] In one embodiment, the gradient update module is further used to determine whether the number of the first sample sequences that have completed back propagation has reached a preset threshold; if the number of the first sample sequences that have completed back propagation has not reached the preset threshold, then after the second sample sequence is input into the model, the second model parameters are continued to be used for training.

[0113] In one embodiment, the gradient update module is further used to determine whether the value of the counter reaches a preset threshold; wherein, whenever any first sample sequence completes back propagation, the value of the counter is accumulated; if the counter reaches the preset threshold, the first model parameter is determined according to a preset flag bit, and after the first model parameter is updated, the flag bit is inverted; if the counter does not reach the preset threshold, the second model parameter is determined according to the preset flag bit.

[0114] In one embodiment, the acquisition module is used to determine the size of the space occupied by the activation value of the sample sequence according to the number of word units; and determine the number of the first sample sequence to be input into the model according to the size of the space occupied by the activation value.

[0115] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0117] Acquire a first sample sequence;

[0118] Inputting the first sample sequence into the model for training, allocating video memory space to each first sample sequence during the training process, and performing operations on each first sample sequence based on a separate thread;

[0119] When any first sample sequence completes back propagation, the video memory space occupied by the first sample sequence is released, and a new second sample sequence is input into the model for training.

[0120] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: during the training process, video memory space is allocated to each first sample sequence, including:

[0121] Determine a plurality of video memory blocks according to the size of the video memory space, and configure a physical address for each video memory block; wherein each video memory block can accommodate activation values ​​generated by a preset number of word units;

[0122] According to the length and the preset number of the first sample sequences, a virtual block table is constructed in the video memory space for each first sample sequence, and physical addresses corresponding to the plurality of video memory blocks are stored in the virtual block table.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when any first sample sequence completes back propagation, releasing the video memory space occupied by the first sample sequence, including:

[0124] In the case where any first sample sequence completes back propagation, determining a virtual block table of the first sample sequence;

[0125] According to the physical address stored in the virtual block table, the corresponding video memory space is released.

[0126] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: when any first sample sequence completes back propagation, the method further includes:

[0127] The gradient values ​​obtained during the back propagation of each first sample sequence are obtained, and the gradient values ​​are accumulated to obtain a gradient accumulation value.

[0128] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: when any first sample sequence completes back propagation, the method further includes:

[0129] Determine whether the number of the first sample sequence that completes the back propagation reaches a preset threshold;

[0130] If the number of the first sample sequences that complete back propagation reaches a preset threshold, the first model parameters are updated based on the gradient accumulation value, and after the second sample sequence is input into the model, training is performed based on the first model parameters.

[0131] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: when any first sample sequence completes back propagation, the method further includes:

[0132] Determine whether the number of the first sample sequence that completes the back propagation reaches a preset threshold;

[0133] If the number of the first sample sequences that have completed back propagation does not reach a preset threshold, after the second sample sequence is input into the model, the second model parameters are continued to be used for training.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining whether the number of first sample sequences that have completed back propagation reaches a preset threshold; if the number of first sample sequences that have completed back propagation reaches the preset threshold, updating the first model parameters based on the gradient accumulation value; if the number of first sample sequences that have completed back propagation does not reach the preset threshold, continuing to use the second model parameters for training, including:

[0135] Determine whether the value of the counter reaches a preset threshold; wherein, whenever any first sample sequence completes reverse propagation, the value of the counter is accumulated;

[0136] If the counter reaches a preset threshold, the first model parameter is determined according to the preset flag bit, and after the first model parameter is updated, the flag bit is inverted;

[0137] If the counter does not reach the preset threshold, the second model parameter is determined according to the preset flag bit.

[0138] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining a first sample sequence, including:

[0139] Determine the number of words that can be accommodated in the current video memory space;

[0140] Determine the size of the space occupied by the activation value of the sample sequence according to the number of word units;

[0141] The number of first sample sequences to be input into the model is determined according to the size of the space occupied by the activation values.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A model training method based on fine-grained scheduling, characterized in that: include: Acquire a first sample sequence; Inputting the first sample sequences into a model for training, allocating video memory space to each of the first sample sequences during the training process, and performing operations on each of the first sample sequences based on a separate thread; When any first sample sequence completes back propagation, the video memory space occupied by the first sample sequence is released, and a new second sample sequence is input into the model for training.

2. The model training method based on fine-grained scheduling according to claim 1 is characterized in that: Allocating video memory space for each of the first sample sequences during the training process includes: Determine a plurality of video memory blocks according to the size of the video memory space, and configure a physical address for each of the video memory blocks; wherein each of the video memory blocks can accommodate activation values ​​generated by a preset number of word units; According to the length of the first sample sequence and the preset number, a virtual block table is constructed in the video memory space for each of the first sample sequences, and physical addresses corresponding to a plurality of the video memory blocks are stored in the virtual block table.

3. The model training method based on fine-grained scheduling according to claim 2 is characterized in that: When any first sample sequence completes back propagation, releasing the video memory space occupied by the first sample sequence includes: In the case where any first sample sequence completes back propagation, determining a virtual block table of the first sample sequence; According to the physical address stored in the virtual block table, the corresponding video memory space is released.

4. The model training method based on fine-grained scheduling according to claim 1 is characterized in that: In the case where any first sample sequence completes back propagation, the method further includes: The gradient values ​​of each of the first sample sequences obtained during the back propagation are obtained, and the gradient values ​​are accumulated to obtain a gradient accumulation value.

5. The model training method based on fine-grained scheduling according to claim 4 is characterized in that: In the case where any first sample sequence completes back propagation, the method further includes: Determine whether the number of the first sample sequence that completes the back propagation reaches a preset threshold; If the number of the first sample sequences that complete back propagation reaches the preset threshold, updating the first model parameters based on the gradient accumulation value, and performing training based on the first model parameters after inputting the second sample sequence into the model; If the number of the first sample sequences that have completed back propagation does not reach the preset threshold, after the second sample sequence is input into the model, the second model parameters are continuously used for training.

6. The model training method based on fine-grained scheduling according to claim 5 is characterized in that: Determine whether the number of first sample sequences that complete back propagation reaches a preset threshold; If the number of the first sample sequences that complete back propagation reaches the preset threshold, updating the first model parameter based on the gradient accumulation value; If the number of the first sample sequences that have completed back propagation does not reach the preset threshold, the second model parameters are continued to be used for training, including: Determine whether the value of the counter reaches the preset threshold; wherein, whenever any first sample sequence completes reverse propagation, the value of the counter is accumulated; If the counter reaches the preset threshold, determining the first model parameter according to a preset flag bit, and after updating the first model parameter, inverting the flag bit; If the counter does not reach the preset threshold, the second model parameter is determined according to the preset flag bit.

7. The model training method based on fine-grained scheduling according to claim 1 is characterized in that: Obtaining a first sample sequence, including: Determine the number of words that can be accommodated in the current video memory space; Determine the occupied space size of the activation value of the sample sequence according to the number of word units; The number of the first sample sequences to be input into the model is determined according to the size of the space occupied by the activation value.

8. A model training device based on fine-grained scheduling, characterized in that: include: An acquisition module, used for acquiring a first sample sequence; An input module, used for inputting the first sample sequences into a model for training, allocating video memory space to each of the first sample sequences during the training process, and performing operations on each of the first sample sequences based on a separate thread; The scheduling module is used to release the video memory space occupied by any first sample sequence when the back propagation of the first sample sequence is completed, and input the new second sample sequence into the model for training.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images