Training method, reasoning method, device and equipment of large language model

Abstract

The invention provides a training method, a reasoning method, a device and equipment for a large language model. Relates to the technical field of computers. The method comprises the steps of obtaining a training text, wherein the training text comprises a first text fragment which is not labeled and a second text fragment which is labeled as a block text; inputting the training text into a large language model, and processing the second text fragment through a block attention mechanism module in the large language model; processing the first text fragment through a standard attention mechanism module in the large language model; calculating a first loss based on a prediction result of the first text fragment output by the large language model; calculating a second loss based on a prediction result of a second text fragment output by the large language model; determining total loss according to the first loss and the second loss; and optimizing parameters of the large language model based on the total loss to obtain a trained large language model. According to the method, the memory reading frequency is reduced, and the efficiency of large language model reasoning is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, reasoning method, device and equipment for a large language model. Background Art

[0002] Large Language Model (LLM), also known as large model for short, is a type of deep learning model trained on massive text data. They are particularly powerful in the field of Natural Language Processing (NLP). Large models usually have many computing layers and a large number of parameters (possibly billions or even tens of billions), and can understand and generate natural language text. After training, large models can not only generate natural language text through reasoning, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, classification, question answering, translation, and document summarization, with strong generalization ability.

[0003] Currently, during the inference process of a large language model, each time a token is inferred, the key-value cache (KV cache) in the memory needs to be read once, resulting in low inference efficiency of the large language model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a training method, reasoning method, device and equipment for a large language model, so as to improve the reasoning efficiency of the large language model.

[0005] In a first aspect, an embodiment of the present application provides a training method for a large language model, wherein the large language model includes a multi-layer Transformer module, each layer of the Transformer module includes an attention mechanism module and a feedforward neural network module; the attention mechanism module includes a standard attention mechanism module and a block attention mechanism module; the method includes: Acquire a training text, wherein the training text includes an unlabeled first text segment and a second text segment labeled as a block text, and the first text segment and the second text segment are arranged alternately; Input the training text into the large language model, process the second text segment through the block attention mechanism module in the large language model; process the first text segment through the standard attention mechanism module in the large language model; Calculate a first loss based on the prediction result of the first text segment output by the large language model; calculate a second loss based on the prediction result of the second text segment output by the large language model; determine a total loss based on the first loss and the second loss; Based on the total loss, the parameters of the large language model are optimized to obtain a trained large language model.

[0006] The embodiment of the present application trains a large language model by using text segments containing block text, so that the large language model can identify which knowledge is relatively simple, so that in the reasoning process of the large language model, the block attention mechanism can be used to process simple content. The large language model can output multiple tokens at one time, and in the process of outputting the token, it only needs to read the KV cache from the memory once, which reduces the number of memory reads and improves the efficiency of the large language model reasoning.

[0007] In any embodiment, processing the first text segment by a standard attention mechanism module in a large language model includes: The first attention value between each token in the first text segment and the token before the token in the training text is calculated through the standard attention mechanism module.

[0008] The embodiment of the present application performs attention calculation on the first text segment through the standard attention mechanism module. Since the first text segment is considered to be complex content, prediction is performed token by token through the standard attention mechanism module, thereby improving the accuracy of large language model reasoning.

[0009] In any embodiment, processing the second text segment by a block attention mechanism module in a large language model includes: Calculate a second attention value between each token in the second text segment and a token before the token in the second text segment through a block attention mechanism module; The third attention value between the second text segment and the token before the second text segment in the training text is calculated through the block attention mechanism module.

[0010] The embodiment of the present application processes the second text segment based on the block attention mechanism module and calculates the internal attention, so that the large language model can better understand the relevance and dependency of these tokens in the local context; by calculating the external attention, the large language model can capture and utilize a wider range of contextual information, thereby improving the accuracy of the large language model reasoning.

[0011] In any embodiment, after obtaining the second attention value and the third attention value, the method further includes: generating an attention matrix based on the first attention value, the second attention value, and the third attention value; The attention matrix is ​​input into the feedforward neural network module in the large language model, and processed by the feedforward neural network module to obtain the prediction result corresponding to the training text output by the large language model.

[0012] The embodiment of the present application processes the attention matrix through a feedforward neural network module to obtain the prediction result corresponding to the training text for subsequent loss calculation.

[0013] In any embodiment, calculating a first loss based on a prediction result of a first text segment output by a large language model includes: According to the formula Calculate the first loss; in, For the first loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the i-th token in the training text; For the known Under the premise, The probability of a token appearing; The first text segment.

[0014] The embodiment of the present application calculates the loss through the cross entropy loss function, which facilitates the optimization of the internal parameters of the large language model.

[0015] In any embodiment, calculating the second loss based on the prediction result of the second text segment output by the large language model includes: According to the formula Calculate the block loss between the second text segment and the token in the training text that precedes the first text segment; According to the formula Calculate the intra-block loss between each token in the second text segment and the token before the token in the second text segment; The sum of the block loss and the intra-block loss is taken as the second loss; in, For block loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the i-th token in the training text; For the known The first text fragment under the premise Probability of occurrence; is the intra-block loss; For the second text segment, in the known Prerequisites Probability of occurrence; is the first text fragment; The second text segment.

[0016] In the embodiment of the present application, the intra-block loss focuses on the relevance of tokens within the second text segment, which can accurately evaluate the performance of the model when processing local contexts, while the block loss considers the relevance between the second text segment and the broader context, which helps to evaluate the model's ability to capture global context information. The dual constraints help improve the overall performance of the model when processing complex text tasks.

[0017] In any embodiment, calculating the second loss based on the prediction result of the second text segment output by the large language model includes: According to the formula Calculate the second loss; in, For the second loss; The internal parameters of the large language model are Calculate the conditional probability obtained when For the training text Tokens; For the known The first text fragment under the premise Probability of occurrence; is the second text fragment; =0,1,2,…,N; where N is the length of the first text segment, ;in, is the length of the second text segment.

[0018] When calculating the loss of the model, the embodiment of the present application takes into account the loss corresponding to the block text and the loss corresponding to the ordinary sequence text, so that a more accurate loss of the model can be obtained, and then based on the loss, the parameters inside the model can be better optimized.

[0019] In a second aspect, an embodiment of the present application provides a method for reasoning a large language model, including: receiving a processing request, the processing request including data to be processed; The data to be processed is input into the large language model, and the large language model determines to use the standard attention mechanism module and / or the block attention mechanism module in the large language model for processing based on the data to be processed, so as to obtain the inference result output by the large language model; The large language model is obtained by training using the method described in the first aspect.

[0020] In a third aspect, an embodiment of the present application provides a training device for a large language model, wherein the large language model includes a multi-layer Transformer module, each layer of the Transformer module includes an attention mechanism module and a feedforward neural network module; the attention mechanism module includes a standard attention mechanism module and a block attention mechanism module; the method includes: A text acquisition module, used for acquiring training text, wherein the training text includes a first text segment that is not labeled and a second text segment that is labeled as a block text, and the first text segment and the second text segment are arranged alternately; A processing module, configured to input the training text into a large language model, process the second text segment through a block attention mechanism module in the large language model, and process the first text segment through a standard attention mechanism module in the large language model; a loss calculation module, configured to calculate a first loss based on a prediction result of the first text segment output by the large language model; calculate a second loss based on a prediction result of the second text segment output by the large language model; and determine a total loss based on the first loss and the second loss; A parameter optimization module is used to optimize the parameters of the large language model based on the total loss to obtain a trained large language model.

[0021] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect or the second aspect by calling the program instructions.

[0022] In a fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method of the first aspect or the second aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, execute the method of the first aspect or the second aspect.

[0024] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a large language model architecture provided in an embodiment of the present application; Figure 2 A flowchart of a method for training a large language model provided in an embodiment of the present application; Figure 3 The mask image of attention provided in the embodiment of the present application; Figure 4 A flowchart of a large language model reasoning method provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a large language model training device provided in an embodiment of the present application; Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0029] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the token " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0032] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0033] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0034] In order to improve the processing efficiency of large language models, the operation of large language models (LLM) is usually based on the combination of GPU and CPU. GPU is a hardware device specially used for processing graphics and image calculations. It was originally developed to accelerate graphics rendering and was later developed to perform large-scale calculations. GPU can provide many parallel processing cores, has strong parallel processing capabilities, is good at performing parallel computing tasks, and can perform most of the reasoning calculation parts of LLM. The CPU is responsible for managing the operating system, processing input / output requests, and performing tasks related to the user interface. The CPU and GPU can work together through the interaction process. The CPU performs calculations based on CPU memory, which is the random access memory (RAM) that interacts directly with the CPU. The GPU performs calculations based on video memory, which is a graphics card that interacts directly with the GPU and is a dedicated memory.

[0035] Time-Per-Output-Token is one of the indicators for evaluating the inference performance of large language models. TBT represents the average delay of each output token from the time the first token is requested to be output to the end of the output. This indicator directly reflects the response speed of the model when generating continuous text, which is crucial for scenarios such as chat applications and real-time translation. A lower TBT means that users can receive responses from the model faster, thereby improving the user experience.

[0036] In the process of large language model inference, a token is a basic unit in the text. This unit can be a character, word, phrase, symbol, or any other form of text fragment. The definition of a token may vary in different contexts, but they are usually the basic unit of text analysis and processing. A token is an unsigned integer obtained by effectively segmenting or wording a text token string and mapping it one by one.

[0037] It should be noted that the big model can process a variety of data types including images and text. Therefore, the data to be processed input into the big model can be an image, a text, or a combination of an image and text. The processing request is a request to be processed for requesting the big model to perform inference calculations. The processing request can be sent to the big model through user operations or through some program processing. This application does not limit the above content.

[0038] The currently popular large models based on the autoregressive Transformer architecture, such as Bailing, LLaMA, Qwen, GPT, and GLM, are a neural network architecture based on the Self-Attention mechanism. The large models under this architecture mainly include two stages of calculation during reasoning, namely the prefill stage and the decode stage, which are executed serially.

[0039] The Prefill stage includes the following processing: After the user inputs the data to be processed into the LLM, the LLM self-encodes the data to be processed, generates attention information for the data to be processed, such as generating the KV cache of the token in the data to be processed, caches the inference result data into the GPU memory, and generates the first token. The inference result data is used in the Decode stage. When the large language model contains N computing layers, the Prefill stage needs to perform the calculations of the N computing layers in sequence to obtain the KV cache data of each token in each computing layer.

[0040] The Decode stage refers to the process in which LLM actually generates the text to be output. In this stage, given the context information (i.e., attention information) of the Prefill stage, the model continues to run reasoning, and then recalculates through N computing layers to generate tokens for the subsequent output text one by one. Each token of the output text needs to be calculated through N computing layers until the reasoning ends. During the execution of the Decode stage, the KV cache data cached in the video memory in the Prefill stage needs to be accessed in the reasoning calculation of each computing layer, and the attention information (Key, Value data) of the newly generated Token in the Decode stage will be spliced ​​with the Keys and Values ​​data cached in the Prefill stage for reasoning.

[0041] In the process of Attention calculation in a large model, in order to reduce the calculation operation, the attention information (including Key and Value) of the generated Token will be cached to form an occupied video memory space, which is KV cache. KVcache will occupy a large amount of GPU video memory, thereby using video memory space to exchange for calculation time.

[0042] In the decoding stage of large model reasoning, for a single user, the main calculation is the general matrix-vector multiplication GEMV, and the hardware computing power and bandwidth are relatively large, resulting in a waste of computing resources. In order to effectively utilize the high computing power resources of the tensor core in the GPU, large language model service providers usually adopt multi-user mode to convert the GEMV operation of a single user into the GEMM operation of multiple users. However, multi-user large model reasoning, especially in long context scenarios, often brings about the problem of insufficient HBM storage of the GPU, resulting in high TBT. Especially in the current scenario where strong reasoning of large models and the use of agents are all long context, it becomes very important to improve the GPU utilization in single-user mode.

[0043] Based on this, the embodiment of the present application provides a training method, reasoning method, device and equipment for a large language model. The embodiment of the present application makes improvements in the architecture and training method of the large language model. Figure 1 A schematic diagram of a large language model architecture provided in an embodiment of the present application is shown in FIG. Figure 1 As shown in the figure, the large language model adopts the decoder-only architecture of Transformer, that is, the large language model includes L stacked Transformer modules, and the value of L can be set according to the actual situation. Each Transformer module includes an attention mechanism module and a feedforward network module FFN. The attention mechanism module includes a standard attention mechanism module and a block attention mechanism module. It should be noted that Figure 1There is no distinction between the standard attention mechanism module and the block attention mechanism module. In terms of structure, the standard attention mechanism module and the block attention mechanism module are the same, but the internal parameters are different. Figure 1 The attention mechanism module shown in FIG. 1 schematically illustrates the architecture of a standard attention mechanism module and / or a block attention mechanism module. It includes three different weight matrices ( ) to perform linear transformation to generate the corresponding query vector Q, key vector K and value vector V, and then use the Softmax function to get the attention weight . Among them, during the training process, the standard attention mechanism module is used to calculate the attention score between the current token and the token before the current token in the training text. The block attention mechanism module is used to calculate the attention score between each token in the current text block in the training text and the token before the token in the current text block, and calculate the attention score between the current text block and the token before the current text block in the training text. Therefore, the training text in the embodiment of the present application includes ordinary sequence text and block text divided into text blocks. In a specific implementation, for a continuous text, a certain continuous text in the text can be regarded as a text block.

[0044] The feedforward neural network module (FFN) consists of two fully connected layers, the first fully connected layer expands the dimension of the input (for example, from 512 dimensions to 2048 dimensions), followed by an activation function (usually ReLU or GELU), and then a second fully connected layer that reduces the dimension from the expanded dimension back to the original dimension (for example, from 2048 dimensions to 512 dimensions).

[0045] The large language model also includes an input layer and an output layer. The input layer includes an embedding layer for vectorizing the input content; the output layer includes a fully connected layer .

[0046] Figure 2 A flow chart of a method for training a large language model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the large language model is executed by a GPU and a CPU. The method includes: Step 201: obtaining a training text, wherein the training text includes a first text segment that is not labeled and a second text segment that is labeled as a block text, and the first text segment and the second text segment are arranged alternately; Step 202: input the training text into the large language model, process the second text segment through the block attention mechanism module in the large language model; process the first text segment through the standard attention mechanism module in the large language model; Step 203: Calculate a first loss based on the prediction result of the first text segment output by the large language model; calculate a second loss based on the prediction result of the second text segment output by the large language model; determine a total loss based on the first loss and the second loss; Step 204: Optimize the parameters of the large language model based on the total loss to obtain a trained large language model.

[0047] The following is a detailed description of each of the above steps.

[0048] In step 201, the training text may be some text from the Internet or customized, and the training text includes a first text segment that is not labeled and a second text segment that is labeled as a block text. The first text segment may include one or more words. The second text segment may include multiple words in most cases. The first text segment and the second text segment are alternately arranged to form a continuous training text.

[0049] For ease of understanding, the following example is given: The training text is: On a quiet morning, "the sun shines through the thin clouds and sprinkles on the earth", and the breeze gently blows through the treetops, bringing a faint fragrance of flowers. Birds sing melodious songs on the branches, as if welcoming the arrival of a new day. The mountains in the distance are looming in the morning mist, "giving people a hazy beauty. The farmers in the fields began their day's work. They bent over and carefully cultivated the fields full of dew." Although life is ordinary, it is full of warmth and hope. Everyone is silently working for their dreams, "looking forward to a better future." In the above training text, the text in "《》" is the second text segment, and the others are the first text segments. It can be understood that which text is divided into text blocks can be manually annotated according to actual needs. For example, simple information or parts with low information density can be annotated as text blocks. It can be understood that text with high information density refers to content that contains more abstract and complex concepts or requires reasoning. For example, technical articles, academic papers, complex conversations, etc., these texts usually require the model to understand a lot of contextual information, reasoning ability, and detailed language understanding. Information entropy can be used to quantify the complexity of the text. Texts with high information entropy tend to have more variables and potential ways of understanding, usually corresponding to higher information density. Texts with high information density usually contain more diverse vocabulary and complex syntactic structures. Such texts will have higher information entropy. On the contrary, it is called low information density.

[0050] In order to enable the large language model to identify which are text blocks and which are ordinary continuous texts, identifiers can be set at the beginning and end of the text block, for example, <|start block|> can be used to identify the beginning of the text block, and <|endblock|> can be used to identify the end of the text block. It should be noted that the start and end characters for identifying the block text can also be other customized symbols, and the embodiments of the present application do not specifically limit this.

[0051] In step 202, the training text is input into the large language model to be trained, the standard attention mechanism module in the large language model processes the first text segment, and the block attention module processes the second text segment. The specific processing process is described in detail in the subsequent embodiments. The standard attention mechanism module is the main module used to generate tokens one by one, which can be a self-attention mechanism module, a linear attention mechanism module, etc.

[0052] In step 203, when the prediction result of the first text segment output by the large language model is predicted and outputted token by token, and when the prediction result of the second text segment output by the large language model is outputted as a whole block, that is, the prediction result of a whole block corresponding to the second text segment is outputted. It should be noted that in the reasoning process of the large language model, if the prediction output is token by token, each time a token is output, it is necessary to access the KV cache from the memory, and if some simple texts are divided into blocks and output by blocks, the output of the block text only needs to access the memory once. Therefore, the reason why some text segments in the training text are divided into text blocks in the embodiment of the present application is to allow the large language model to learn which content can be output by block and which content needs to be output by token. For example: through a large amount of training, the large language model can master some knowledge. In the reasoning process, for the large language model, relatively simple content can be output by block, and relatively difficult content can be output by token. This improves the efficiency of model reasoning while taking into account the accuracy of the output content.

[0053] Therefore, when calculating the loss, the first loss corresponding to the first text segment and the second loss corresponding to the second text segment are accumulated as the total loss of the large language model.

[0054] In step 204, after the total loss is obtained, the internal parameters in the large language model are reversely optimized using the total loss to obtain an optimized large language model.

[0055] It should be noted that in order to improve the reasoning performance of the large language model, a large number of training texts can be used to perform multiple training according to the method of step 201 to step 204. The condition for the end of the training can be that the iterative training reaches a preset number of times, or the loss change rate meets the condition, etc.

[0056] The embodiment of the present application trains a large language model by using text segments containing block text, so that the large language model can identify which knowledge is relatively simple, so that in the reasoning process of the large language model, the block attention mechanism can be used to process simple content. The large language model can output multiple tokens at one time, and in the process of outputting the token, it only needs to read the KV cache from the memory once, which reduces the number of memory reads and improves the efficiency of the large language model reasoning.

[0057] Based on the above embodiment, processing the first text segment by using the standard attention mechanism module in the large language model includes: The first attention value between each token in the first text segment and the previously output token is calculated through the standard attention mechanism module.

[0058] In the specific implementation process, the standard attention mechanism module can be a self-attention mechanism, which is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. It allows the model to dynamically adjust the degree of attention to each element when processing sequence data, thereby capturing the complex dependencies within the sequence. The core of the self-attention mechanism is that it does not rely on external information, but interacts and integrates information between elements within the sequence. It should be noted that the standard attention mechanism module can also be a linear attention mechanism module, etc.

[0059] After the first text segment is input into the large language model, the first text segment can be preprocessed, for example, each token in the first text segment is converted into a vector representation. This is usually achieved through an embedding layer to map the token to a vector in a high-dimensional space.

[0060] For each token vector in the input sequence, three different weight matrices are used ( ) is linearly transformed to generate the corresponding query vector Q, key vector K and value vector V. These weight matrices are learnable parameters during the training process.

[0061] For each token in the sequence, the dot product is calculated using its query vector Q and all key vectors K to obtain an attention score. In order to stabilize training and improve generalization, the attention score is usually divided by the square root of the key vector dimension (i.e., the scaling factor). Apply the Softmax function to the scaled attention score to obtain the attention weight. Softmax ensures that the sum of all output weights is 1, so that the model can learn the importance of each token pair. Use the attention weights obtained above to perform a weighted summation on the value vector V to generate the final output vector. This step can effectively integrate the information within the sequence so that the output representation of each token contains the contextual information of the entire sequence.

[0062] Apply the above process to all tokens in the sequence to construct the first attention value.

[0063] The embodiment of the present application performs attention calculation on the first text segment through the standard attention mechanism module. The first text segment is considered to be complex content. Therefore, prediction is performed token by token through the standard attention mechanism module, thereby improving the accuracy of large language model reasoning.

[0064] Based on the above embodiment, the second text segment is processed by the block attention mechanism module in the large language model, including: Calculate a second attention value between each token in the second text segment and a token before the token in the second text segment through a block attention mechanism module; The third attention value between the second text segment and the token before the second text segment in the training text is calculated through the block attention mechanism module.

[0065] In a specific implementation process, first, a second text segment is identified and extracted from the entire training text, and the second text segment includes a series of tokens. Since the first and last parts of the second text segment are both provided with identifiers, the second text segment can be extracted based on the identifiers.

[0066] The block attention mechanism module is designed to reduce the number of times the KV cache is read from the memory when processing long sequence data. That is, during the model inference process, the generation of the second text segment only requires reading the KV cache from the memory once. In addition, the large language model outputs the second text segment at one time.

[0067] The standard attention mechanism module calculates the attention of two tokens per token to form a standard lower triangular attention matrix. Different from the standard attention mechanism module, the block attention mechanism module batch calculates the attention of two tokens in the current token block and the attention of tokens in the block and previous tokens. The attention matrix formed is not a standard lower triangular attention matrix. The attention matrix is ​​obtained by attention calculation with mask. Figure 3 The mask image of the attention provided in the embodiment of the present application is as follows: Figure 3 As shown. The value on the blank box is 0, and the value on the shaded box is 1. The training text is the first text segment and the second text segment arranged alternately. Therefore, from top to bottom, the first 4 tokens are the first text segment. The traditional attention mechanism transformer decoder-only architecture is used, and its attention matrix is ​​lower triangular. When calculating the 5th to 7th tokens as the second text segment, since the 5th to 7th tokens are output together, a method similar to BERT is used. Therefore, the attention value of the second text segment forms a 3x3 matrix on 5-7, and the mask value of the 3x3 attention matrix is ​​all 1, which is a non-lower triangular matrix attention form; when the second text segment and the first text segment calculate each other's attention, due to the use of the decoder-only architecture, the overall attention matrix formed by the second text block and the first text block is lower triangular, and it is non-lower triangular at the position of the block text output. After obtaining the second text segment, the block attention mechanism module calculates the attention value within the block and the attention value outside the block. The so-called in-block attention value refers to the attention value between any two tokens in the second text segment. For example: the second text segment includes token1, token2, and token3. The block attention mechanism module calculates the attention value between token1 and token2, the attention value between token1 and token3, the attention value between token2 and token1, the attention value between token2 and token3, the attention value between token3 and token1, and the attention value between token3 and token2.

[0068] The out-of-block attention value refers to the attention value between the second text segment and the token that appears before the second text segment in the training text. For example: the training text includes token1, token2, token3 and token4. Among them, token1 and token2 are ordinary continuous texts, that is, the first text segment. Token3 and token4 are a text block, that is, the second text segment. When calculating the out-of-block attention value, it is necessary to calculate the attention value between token3 and token4 as a whole and token1, and the attention value between token3 and token4 as a whole and token2.

[0069] It should be noted that the method of calculating the values ​​of each element in the matrix in the second attention value and the third attention value is similar to the method of calculating the values ​​of each element in the first attention value. They are all calculated using the query vector Q, key vector K and value vector V, but the tokens involved in the calculation are different, which will not be repeated here.

[0070] The embodiment of the present application processes the second text segment based on the block attention mechanism module, calculates the internal attention, so that the large language model can better understand the relevance and dependency of these tokens in the local context; by calculating the external attention, the large language model can capture and utilize a wider range of contextual information. The accuracy of the large language model reasoning is improved.

[0071] Based on the above embodiment, after obtaining the second attention value and the third attention value, the method further includes: generating an attention matrix based on the first attention value, the second attention value, and the third attention value; The attention matrix is ​​input into the feedforward neural network module in the large language model, and processed by the feedforward neural network module to obtain the prediction result corresponding to the training text output by the large language model.

[0072] In a specific implementation process, after obtaining the first attention value, the second attention value and the third attention value, a corresponding attention matrix can be generated, where the attention matrix is ​​the same as the above Figure 3 The mask image is similar, except that the element value corresponding to the blank box is 0, and the element value of the shadow box is the corresponding attention value.

[0073] After obtaining the attention matrix, the attention matrix is ​​input into the feedforward neural network in the large language model. The feedforward neural network module usually includes multiple layers of linear transformation and nonlinear activation function. Each layer receives the output of the previous layer, and generates a new output after linear transformation and nonlinear activation. The feedforward neural network module provided in the embodiment of the present application may include two layers of linear transformation and two layers of nonlinear activation function. It is understandable that the specific number of layers of linear transformation and nonlinear activation function included in the feedforward neural network module can be adjusted according to actual conditions, and the embodiment of the present application does not specifically limit this. In the Transformer architecture, the feedforward neural network model is usually located after the attention mechanism module, which is used to further process the vector representation of the output of the attention mechanism module.

[0074] After the feedforward neural network module receives the attention matrix, it performs a series of linear transformations and nonlinear activation operations on it. These operations are designed to capture higher-order features in the input sequence and generate richer representations. After being processed by the feedforward neural network module, the output will contain a deeper understanding and analysis of the input sequence.

[0075] At the output end of the feedforward neural network module, an output layer is usually connected. This output layer is responsible for converting the output of the feedforward neural network into the final prediction result. The output layer may contain components such as linear transformation and Softmax function, which are used to convert the output of the neural network into probability distribution or other forms of prediction results.

[0076] The embodiment of the present application processes the content output by the attention mechanism module based on the feedforward neural network module to obtain a prediction result, which is used for loss calculation to provide a basis for subsequent optimization of the internal parameters of the large language model.

[0077] Based on the above embodiments, the loss function structure of the embodiments of the present application includes the classic and commonly used LLM training loss function, the intra-block loss function and the block loss function.

[0078] Among them, the LLM training loss function is calculated based on the prediction result of the first text segment, that is, the first loss. The intra-block loss function is based on the loss between each token in the second text segment and the token before the token in the second text segment, that is, the intra-block loss. The block loss function is based on the loss between the second text segment and the tokens between the second text segments in the training text, that is, the block loss. The following is an introduction to each loss function: The calculation formula of the first loss is as follows:

[0079] The calculation formula of intra-block loss is as follows:

[0080] The calculation formula of block loss is as follows:

[0081] in, For the first loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the i-th token in the training text; For the known Under the premise, The probability of a token appearing; For block loss; For the known The first text fragment under the premise Probability of occurrence; is the intra-block loss; For the second text segment, in the known Prerequisites Probability of occurrence; is the first text fragment; The second text segment.

[0082] The second loss is the sum of the block loss and the intra-block loss. After obtaining the first loss and the second loss, the sum of the first loss and the second loss is taken as the total loss. That is, the total loss = + + .

[0083] It should be noted that there may be multiple first text segments and multiple second text segments in the training text, the first loss is the sum of the losses of the multiple first text segments, and the second loss is the sum of the losses of the multiple second text segments.

[0084] After obtaining the total loss, the internal parameters of the large language model are optimized in reverse based on the total loss to obtain the optimized large language model.

[0085] After completing the model training, the large language model can be deployed in a computer or server. During the deployment, it can be compatible with the commonly used low-bit quantization accuracy. The low-bit quantization methods include: LLM.int8(), soomthquant and other commonly used LLM quantization technologies; as well as the deployment of classic large models, supporting reasoning frameworks such as Vllm and SGLang; supporting technologies such as PD separation at the cluster deployment level.

[0086] It should be noted that the above training method can be used to train a large language question-answering model, that is, input a question to the large language model, and the large language model can return the corresponding answer. It can also be used to train a multimodal understanding model.

[0087] On the basis of the above embodiment, calculating the second loss based on the prediction result of the second text segment output by the large language model includes: According to the formula Calculate the second loss.

[0088] in, For the second loss; The internal parameters of the large language model are Calculate the conditional probability obtained when For the training text Tokens; For the known The first text fragment under the premise Probability of occurrence; is the second text fragment; =0,1,2,…,N; where N is the length of the first text segment, ;in, is the length of the second text segment.

[0089] Therefore, based on the above embodiment, it can be seen that the second loss in the embodiment of the present application only considers the block loss. The total loss of the model = + , is the first loss, and its calculation formula can be found in the above embodiment, which will not be repeated here. Compared with the solution in the above embodiment that takes into account the block loss and the intra-block loss, the solution in the embodiment of the present application reduces the amount of calculation.

[0090] When calculating the loss of the model, the embodiment of the present application takes into account the loss corresponding to the block text and the loss corresponding to the ordinary sequence text, so that a more accurate loss of the model can be obtained, and then based on the loss, the parameters inside the model can be better optimized.

[0091] Based on the large language model trained by the above large language model training method, the following reasoning method can be performed: Figure 4 A flowchart of a large language model reasoning method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes: Step 401: receiving a processing request, the processing request including data to be processed; Step 402: Input the data to be processed into the large language model. The large language model determines whether to use the standard attention mechanism module and / or the block attention mechanism module in the large language model for processing based on the data to be processed to obtain the inference result output by the large language model.

[0092] In the specific implementation process, the data to be processed can be text, numbers, images, files or other formats, depending on the requirements of the application scenario. For example, in the scenario of natural language processing, the data to be processed can be a piece of text; in the scenario of image processing, the data to be processed can be one or more images. Of course, the data to be processed can also be multimodal data, for example, it can include images and text, files and text, images, files and text, etc.

[0093] Processing requests can be submitted by users through the user interface or sent by other systems or services through the API interface.

[0094] The received data to be processed is input into the large language model obtained by training according to the training method provided in the above embodiment. The large language model performs reasoning based on the data to be processed. During the reasoning process, the large language model can identify whether it is necessary to use the block attention mechanism module for reasoning or the standard attention mechanism module for reasoning. If the block attention mechanism module is used for reasoning, the large language model can output multiple tokens at the same time; if the standard attention mechanism module is used for reasoning, the large language model outputs tokens one by one. Therefore, during the reasoning process, the large language model may only use the standard attention mechanism module for reasoning, or may only use the block attention mechanism module for reasoning, or may use both the standard attention mechanism module and the block attention mechanism module. For the case where both the standard attention mechanism module and the block attention mechanism module are used, among the tokens output by the large language model, some tokens are output one by one, and some continuous tokens are output at one time. It should be noted that the above description of only using the block attention mechanism module and only using the standard attention mechanism module refers to the attention mechanism module. In the actual reasoning process, in addition to the attention mechanism module (block attention mechanism module and / or standard attention mechanism module), it is also necessary to use the processing of modules such as the feedforward neural network module.

[0095] The embodiment of the present application utilizes the large language model reasoning process to effectively locate when the block attention mechanism module needs to be used for processing, generate text in batches, and improve the efficiency of model reasoning.

[0096] Figure 5This is a schematic diagram of a large language model training device provided in an embodiment of the present application. The device may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 2 The method embodiment corresponds to and can be executed Figure 2 The various steps involved in the method embodiment and the specific functions of the device can be found in the above description. To avoid repetition, detailed description is appropriately omitted here. The device includes: a text acquisition module 501, a processing module 502, a loss calculation module 503 and a parameter optimization module 504, wherein: The text acquisition module 501 is used to acquire training text, wherein the training text includes a first text segment that is not labeled and a second text segment that is labeled as a block text, and the first text segment and the second text segment are arranged alternately; The processing module 502 is used to input the training text into the large language model, process the second text segment through the block attention mechanism module in the large language model; and process the first text segment through the standard attention mechanism module in the large language model; The loss calculation module 503 is used to calculate a first loss based on the prediction result of the first text segment output by the large language model; calculate a second loss based on the prediction result of the second text segment output by the large language model; and determine a total loss based on the first loss and the second loss; The parameter optimization module 504 is used to optimize the parameters of the large language model based on the total loss to obtain a trained large language model.

[0097] Based on the above embodiment, the processing module 502 is specifically used for: A first attention value between each token in the first text segment and a token preceding the token in the training text is calculated by the standard attention mechanism module.

[0098] Based on the above embodiment, the processing module 502 is specifically used for: Calculating a second attention value between each token in the second text segment and a token before the token in the second text segment through the block attention mechanism module; A third attention value between the second text segment and a token preceding the second text segment in the training text is calculated by the block attention mechanism module.

[0099] Based on the above embodiment, the processing module is further used for: generating an attention matrix based on the first attention value, the second attention value, and the third attention value; The attention matrix is ​​input into a feedforward neural network module in the large language model, and is processed by the feedforward neural network module to obtain a prediction result corresponding to the training text output by the large language model.

[0100] Based on the above embodiment, the loss calculation module 503 is specifically used for: According to the formula Calculate and obtain the first loss; in, is the first loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the i-th token in the training text; For the known Under the premise, The probability of a token appearing; This is the first text segment.

[0101] Based on the above embodiment, the loss calculation module 503 is specifically used for: According to the formula Calculating a block loss between the second text segment and a token in the training text that precedes the first text segment; According to the formula Calculating the intra-block loss between each token in the second text segment and the token before the token in the second text segment; taking the sum of the block loss and the intra-block loss as the second loss; in, is the block loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the i-th token in the training text; For the known Under the premise that the first text fragment Probability of occurrence; is the loss within the block; For the second text fragment, in the known Prerequisites Probability of occurrence; is the first text segment; This is the second text segment.

[0102] Based on the above embodiment, the loss calculation module 503 is specifically used for: According to the formula Calculate the second loss; in, For the second loss; The internal parameters of the large language model are Calculate the conditional probability obtained when For the training text Tokens; For the known The first text fragment under the premise Probability of occurrence; is the second text fragment; =0,1,2,…,N; where N is the length of the first text segment, ;in, is the length of the second text segment.

[0103] Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a processor (processor) 601, a memory (memory) 602 and a bus 603; wherein, The processor 601 and the memory 602 communicate with each other via the bus 603; The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a training text, wherein the training text includes an unlabeled first text segment and a second text segment labeled as a block text, and the first text segment and the second text segment are arranged alternately; inputting the training text into a large language model, and processing the second text segment through a block attention mechanism module in the large language model; processing the first text segment through a standard attention mechanism module in the large language model; calculating a first loss based on a prediction result of the first text segment output by the large language model; calculating a second loss based on a prediction result of the second text segment output by the large language model; determining a total loss based on the first loss and the second loss; and optimizing the parameters of the large language model based on the total loss to obtain a trained large language model.

[0104] The processor 601 may be an integrated circuit chip with signal processing capabilities. The processor 601 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] The memory 602 may include, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0106] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a training text, wherein the training text includes a first unlabeled text segment and a second text segment labeled as a block text, and the first text segment and the second text segment are arranged alternately; inputting the training text into a large language model, and processing the second text segment through a block attention mechanism module in the large language model; processing the first text segment through a standard attention mechanism module in the large language model; calculating a first loss based on a prediction result of the first text segment output by the large language model; calculating a second loss based on a prediction result of the second text segment output by the large language model; determining a total loss based on the first loss and the second loss; optimizing the parameters of the large language model based on the total loss to obtain a trained large language model.

[0107] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a training text, wherein the training text includes a first unlabeled text segment and a second text segment labeled as a block text, and the first text segment and the second text segment are arranged alternately; inputting the training text into a large language model, and processing the second text segment through a block attention mechanism module in the large language model; processing the first text segment through a standard attention mechanism module in the large language model; calculating a first loss based on a prediction result of the first text segment output by the large language model; calculating a second loss based on a prediction result of the second text segment output by the large language model; determining a total loss based on the first loss and the second loss; optimizing the parameters of the large language model based on the total loss to obtain a trained large language model.

[0108] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0109] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0111] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0112] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a large language model, characterized in that: The large language model includes a multi-layer Transformer module, and each layer of the Transformer module includes an attention mechanism module and a feedforward neural network module; The attention mechanism module includes a standard attention mechanism module and a block attention mechanism module; the method includes: Acquire a training text, wherein the training text includes an unlabeled first text segment and a second text segment labeled as a block text, and the first text segment and the second text segment are arranged alternately; Input the training text into a large language model, process the second text segment through a block attention mechanism module in the large language model; process the first text segment through a standard attention mechanism module in the large language model; Calculating a first loss based on the prediction result of the first text segment output by the large language model; calculating a second loss based on the prediction result of the second text segment output by the large language model; determining a total loss based on the first loss and the second loss; Parameters of the large language model are optimized based on the total loss to obtain a trained large language model.

2. The method according to claim 1, characterized in that: The processing of the first text segment by a standard attention mechanism module in the large language model includes: A first attention value between each token in the first text segment and a token preceding the token in the training text is calculated by the standard attention mechanism module.

3. The method according to claim 2, characterized in that The processing of the second text segment by the block attention mechanism module in the large language model includes: Calculating a second attention value between each token in the second text segment and a token before the token in the second text segment through the block attention mechanism module; A third attention value between the second text segment and a token preceding the second text segment in the training text is calculated by the block attention mechanism module.

4. The method according to claim 3, characterized in that After obtaining the second attention value and the third attention value, the method further includes: generating an attention matrix based on the first attention value, the second attention value, and the third attention value; The attention matrix is ​​input into a feedforward neural network module in the large language model, and is processed by the feedforward neural network module to obtain a prediction result corresponding to the training text output by the large language model.

5. The method according to claim 1, characterized in that The calculating a first loss based on the prediction result of the first text segment output by the large language model includes: According to the formula Calculate and obtain the first loss; in, is the first loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the first Tokens; For the known Under the premise, The probability of a token appearing; is the first text segment; =0,1,2,…,N, where N is the length of the first text segment.

6. The method according to any one of claims 1 to 5, characterized in that: The calculating a second loss based on the prediction result of the second text segment output by the large language model includes: According to the formula Calculating a block loss between the second text segment and a token in the training text that precedes the first text segment; According to the formula Calculating the intra-block loss between each token in the second text segment and the token before the token in the second text segment; taking the sum of the block loss and the intra-block loss as the second loss; in, is the block loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the first Tokens; For the known Under the premise that the first text fragment Probability of occurrence; is the loss within the block; For the second text fragment, in the known Prerequisites Probability of occurrence; is the first text segment; is the second text segment; =0,1,2,…,N; wherein N is the length of the first text segment, ;in, is the length of the second text segment.

7. The method according to any one of claims 1 to 5, characterized in that: The calculating a second loss based on the prediction result of the second text segment output by the large language model includes: According to the formula Calculate and obtain the second loss; in, for the second loss; The internal parameters of the large language model are Calculate the conditional probability obtained when is the first Tokens; For the known Under the premise that the first text fragment Probability of occurrence; is the second text segment; =0,1,2,…,N; wherein N is the length of the first text segment, ;in, is the length of the second text segment.

8. A method for reasoning about a large language model, characterized in that: include: receiving a processing request, wherein the processing request includes data to be processed; Inputting the data to be processed into a large language model, the large language model determining, based on the data to be processed, to adopt a standard attention mechanism module and / or a block attention mechanism module in the large language model for processing, and obtaining an inference result output by the large language model; Wherein, the large language model is obtained by training using the method described in any one of claims 1-7.

9. A large language model training device, characterized in that: The large language model includes a multi-layer Transformer module, and each layer of the Transformer module includes an attention mechanism module and a feedforward neural network module; The attention mechanism module includes a standard attention mechanism module and a block attention mechanism module; the device includes: A text acquisition module, used for acquiring training text, wherein the training text includes a first text segment that is not labeled and a second text segment that is labeled as a block text, and the first text segment and the second text segment are arranged alternately; A processing module, configured to input the training text into a large language model, process the second text segment through a block attention mechanism module in the large language model, and process the first text segment through a standard attention mechanism module in the large language model; a loss calculation module, configured to calculate a first loss based on a prediction result of the first text segment output by the large language model; calculate a second loss based on a prediction result of the second text segment output by the large language model; and determine a total loss based on the first loss and the second loss; A parameter optimization module is used to optimize the parameters of the large language model based on the total loss to obtain a trained large language model.

10. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

12. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images