Language model using method for sharing memory matrix

Abstract

The invention relates to the technical field of computer language processing, in particular to a language model using method for sharing a memory matrix. According to the method, a Key matrix and a Value matrix are obtained through memory through a cyclic iteration structure, and meanwhile, the memory of each layer in the process of reasoning the current token is ensured to be the same and is equivalent to sharing of a key tensor and a value tensor, so that the effect of reducing the cache is realized, and the computing power resource is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer language processing, and in particular to a method for using a language model of a shared memory matrix. Background Art

[0002] In the field of natural language processing, the rise of large language models (LLMs) has enabled us to witness significant technological progress in AI. However, the scale and complexity of these models also bring challenges to the efficiency of reasoning. In order to improve this efficiency, the key-value cache (KV Cache) technology came into being and has been widely used in practice. Key-value cache (KV Cache) is an efficient data storage mechanism that optimizes access speed by caching key-value pairs of data. During the reasoning process of the language model, some data may be frequently accessed. KV Cache achieves fast data retrieval by storing these repeatedly accessed data in memory, thereby significantly speeding up the reasoning speed. It is worth noting that KV Cache is mainly used in the decoding stage of the model, such as in the decoding process of "decoding only" models (such as GPT-3, LLaMa3, etc.) and "encode-decode" models (such as T5). During the reasoning process of the language model, the model generates the corresponding answer based on the input question. This generation process is carried out step by step, and only one token is generated at a time. The newly generated token is immediately added to the original question to form a new input sequence, which is passed to the model again for the next step of reasoning. This process will be repeated until the model generates a terminator, indicating the completion of the answer. However, in the process of large model reasoning, as the text length increases and the embedding dimension increases, the KV Cache memory usage will also increase rapidly, resulting in computing power consumption. Based on this, the industry proposed MQA (Multi Query Attention) and MHA (Grouped Query Attention), which mainly adopt parameter sharing for each head of multi-head attention. Although the memory usage problem has been alleviated, the effect is still limited. Summary of the invention

[0003] This patent proposes a language model structure with a shared memory matrix, aiming to reduce the problem of excessive cache usage in KV Cache during reasoning of long texts. At the same time, this patent belongs to a small text generation language model.

[0004] The following technical solution is adopted to solve the existing technical problems. A method for using a language model of a shared memory matrix comprises the following steps: S1: Create tokens and define special characters: Collect open source language training sets and use the SentencePiece tool for training, which includes the reserved language model [CLS] text start symbol and [PAD] text filler symbol. The agreed max_len=2048 is the maximum length of the text input to the model, which is also the maximum length of the model output. S2: Construct model: The current training text, denoted as {Text}, is divided into a word list using the tokenize word segmentation tool, and the [CLS] start symbol is added to the front, which is the tokens list input to the model. The tokens list without [CLS] at the front is the actual model output, that is, the input [CLS] needs to be predicted as the first token after the output {Text} is serialized; the memory vector is used as a historical storage, initially None, [CLS] embedding vector, during the training process, the first cycle memory is assigned to the [CLS] embedding vector, and the [CLS] embedding vector plays the role of query. The memory is fully connected through two tensors to play the roles of key tensor and value tensor respectively, and then cross-attention calculation is performed with the query. At the same time, it passes through the feedforward neural network to obtain a new representation of the first-layer embedding vector of the current token. The embedding representation is used as the new tensor representation of the second-layer query, but the memory tensor of each subsequent layer is The key tensor and value tensor obtained by full connection are the same as those of the first layer when training or reasoning the current token, that is, each subsequent layer uses the key tensor and value tensor obtained by the same tensor in memory. In the first loop memory, only the [CLS] embedding vector is stored, and then the tensors key and value are obtained respectively through two full connections. At the same time, the first [CLS] vector plays the role of query tensor, and then cross attention is calculated until multiple layers. In the second loop memory, the [CLS] embedding vector and the first input token vector are concatenated, and then the tensors key and value are obtained respectively through two full connections. The first input token vector plays the role of query tensor, which can be passed through multiple layers, and so on. In the third loop memory, the memory embedding vector of the second loop is concatenated with the second input token vector, and then the tensors key and value are obtained respectively through two full connections. And so on. S3: Reasoning stage: Tokenize [CLS] and the content input by the user and concatenate them into memory. The last token of the user input content plays the role of the query tensor. Calculate the cross attention and feedforward layer until the output to get the vector representation of the first token. Then decode it through full connection to get the first generated token. Then, concatenate the token into the memory tensor through the embedding layer. At the same time, the token plays the role of the query tensor. Repeat the above operations to complete the final model text generation content.

[0005] The beneficial effect of the present invention is that it proposes a solution to the problem of excessive memory usage of KV Cache in the industry during the process of ultra-long text reasoning. This patent uses a cyclic iterative structure to obtain the Key matrix and the Value matrix through memory, while ensuring that the memory of each layer is the same during the reasoning of the current token, which is equivalent to sharing the key tensor and the value tensor, thereby achieving the effect of cache reduction, and thus saving computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A flow chart is used for the network model of the present invention. DETAILED DESCRIPTION

[0007] S1: Create tokens and define special symbols: S1 is an existing technology in the industry, but in order to clearly explain the innovation of this patent, it is supplemented by collecting open source language training sets and using the SentencePiece tool for training, which includes the reserved [CLS] text start symbol and [PAD] text filler symbol of the language model. This patent stipulates that max_len=2048 is the maximum text length of the input model stipulated in this patent, and it is also the maximum length of the model output. However, people who use this patent can train longer models according to their own circumstances.

[0008] S2: Construction model: This is the main innovation of this patent. The model structure is shown in Figure 1 The technical implementation of this patent is roughly consistent with the principles of Llama-like and GPT-like language models in the reasoning stage. In the mainstream large language model reasoning process, the use of KV_Cache technology in super-long text reasoning will gradually increase the Keys and Values ​​caches, occupying a large amount of video memory. Multi-Query Attention and Group-Query Attention achieve resource reduction through parameter sharing. However, this solution is far from enough for super-long texts. The model proposed in this patent is as follows Figure 1This patent transforms the Llama-like training phase into a loop-serial mode, aiming to avoid the problem of difficult optimization of cache content caused by the causal attention matrix in the current large language model. The large language model structure proposed in this patent refers to Figure 1, explained as follows: A certain training text, denoted as {Text}, is divided into a word list using the tokenize word segmentation tool, and the [CLS] start symbol is added at the front, which is the tokens list input to the model. The tokens list without [CLS] at the front is the actual model output, that is, the input [CLS] needs to be predicted as the first token after the output {Text} is serialized. The memory vector is used as the historical storage, and is initially None. The [CLS] embedding vector is assigned to the [CLS] embedding vector in the first loop during training. The [CLS] embedding vector plays the role of query. The memory is fully connected to obtain two tensors that play the roles of key tensor and value tensor respectively, and then performs cross-attention calculation with the query. At the same time, it passes through the feedforward neural network to obtain a new representation of the first-layer embedding vector of the current token. The embedding representation is used as the new tensor representation of the second-layer query. However, the key tensor and value tensor obtained by fully connecting the memory tensor of each subsequent layer are the same as the first-layer representation when training or inferring the current token. That is, each subsequent layer uses the key tensor and value tensor obtained by the same tensor of memory to realize memory tensor sharing. This has obvious advantages for large language models. If a large language model such as GPT has N layers, without introducing other methods, the memory occupancy of this patent will be reduced by 1 / N, so The effect of using this patent for large language models is very obvious. The query tensor of this patent increases with the number of model layers, and the semantic understanding ability continues to increase, which improves the expression ability of the model. The first loop memory only stores the [CLS] embedding vector, and then obtains the tensor key and value respectively through two full connections. At the same time, the first [CLS] vector plays the role of the query tensor, and then calculates the cross attention until multiple layers. The second loop memory concatenates the [CLS] embedding vector and the first input token vector, and then obtains the tensor key and value respectively through two full connections. The first input token vector plays the role of the query tensor, which can pass through multiple layers, and so on. The third loop memory appends the memory embedding vector of the second loop to the second input token vector, and then obtains the tensor key and value respectively through two full connections. And so on. In the process of generating ultra-long texts, this patent avoids the problem of KV Cache cache content gradually increasing with the increase in the number of layers through memory tensor cache sharing, which saves computing power resources. The above content is the main protection point of this patent.

[0009] S3: Reasoning stage: In the reasoning stage, [CLS] and the content input by the user are tokenized and concatenated and stored as memory. The last token of the user input content plays the role of the query tensor. The cross attention and feedforward layer are calculated until the output to obtain the vector representation of the first token. Then, the first token generated is obtained by decoding after full connection. The token is then concatenated to the memory tensor through the embedding layer. At the same time, the token plays the role of the query tensor. Repeat the above operations to complete the final model text generation content.

[0010] Glossary Shared memory matrix: The shared memory matrix mainly memorizes the historical token embedding tensors of the input large language model through caching. For example, to predict the model output of the third position of the language model, the memory matrix needs to cache the three embedding vectors of the model start symbol, the model's first actual token, and the model's second actual token. At the same time, the model inputs the embedding vector of the second actual token, and calculates the cross attention between the embedding vector of the model input's second actual token and the three token embedding vectors in the memory matrix. The memory matrix Memory is equivalent to playing the role of the key tensor and value tensor in the Transformer.

[0011] KV Cache: KV Cache is an efficient reasoning acceleration technology specifically designed for large language models. It builds a rich parameter cache library by retaining and appending the keys and values ​​obtained from historical calculations at each stage of text generation. This cache library enables the model to reuse previous calculation results, effectively reducing redundant calculations in text generation tasks and greatly improving the operating efficiency of the model.

[0012] Language model: A language model is a statistical model whose main goal is to predict the probability of a given sequence (usually a text sequence). In the field of natural language processing (NLP), language models are key core components and are widely used in various tasks such as machine translation, speech recognition, automatic summarization, sentiment analysis, and dialogue systems.

[0013] Transformer: The Transformer language model is a text translation language model based on a multi-head self-attention mechanism. In machine translation tasks, it surpasses previous models that rely on recurrent neural networks (RNNs).

[0014] Token: In language models, tokens usually refer to the smallest unit of text, usually single-word tokens, word tokens, subword tokens, punctuation marks, numbers, etc. Tokens are used to represent part of the model input. Each token has a corresponding number. In the embedding layer of the language model, each token is usually assigned a vector representation. Today, tokens have a broader meaning. Tokens can be abstract tool names or even activity names that people define.

Claims

1. A method for using a language model of a shared memory matrix, characterized in that The steps include: S1: Create tokens and define special characters: Collect open source language training sets and use the SentencePiece tool for training, which includes the reserved language model [CLS] text start symbol and [PAD] text filler symbol. The agreed max_len=2048 is the maximum length of the text input to the model, which is also the maximum length of the model output. S2: Construct model: The current training text, denoted as {Text}, is divided into a word list using the tokenize word segmentation tool, and the [CLS] start symbol is added to the front, which is the tokens list input to the model. The tokens list without [CLS] at the front is the actual model output, that is, the input [CLS] needs to be predicted as the first token after the output {Text} is serialized; the memory vector is used as a historical storage, initially None, [CLS] embedding vector, during the training process, the first cycle memory is assigned to the [CLS] embedding vector, and the [CLS] embedding vector plays the role of query. The memory is fully connected through two tensors to play the roles of key tensor and value tensor respectively, and then cross-attention calculation is performed with the query. At the same time, it passes through the feedforward neural network to obtain a new representation of the first-layer embedding vector of the current token. The embedding representation is used as the new tensor representation of the second-layer query, but the memory tensor of each subsequent layer is The key tensor and value tensor obtained by full connection are the same as those of the first layer when training or reasoning the current token, that is, each subsequent layer uses the key tensor and value tensor obtained by the same tensor in memory. In the first loop memory, only the [CLS] embedding vector is stored, and then the tensors key and value are obtained respectively through two full connections. At the same time, the first [CLS] vector plays the role of query tensor, and then cross attention is calculated until multiple layers. In the second loop memory, the [CLS] embedding vector and the first input token vector are concatenated, and then the tensors key and value are obtained respectively through two full connections. The first input token vector plays the role of query tensor, which can be passed through multiple layers, and so on. In the third loop memory, the memory embedding vector of the second loop is concatenated with the second input token vector, and then the tensors key and value are obtained respectively through two full connections. And so on. S3: Reasoning stage: Tokenize [CLS] and the content input by the user and concatenate them into memory. The last token of the user input content plays the role of the query tensor. Calculate the cross attention and feedforward layer until the output to get the vector representation of the first token. Then decode it through full connection to get the first generated token. Then, concatenate the token into the memory tensor through the embedding layer. At the same time, the token plays the role of the query tensor. Repeat the above operations to complete the final model text generation content.

Images