A language model using method of sharing a memory matrix

Abstract

The application relates to the technical field of computer language processing, in particular to a language model using method of a shared memory matrix. Through a cyclic iteration structure, a Key matrix and a Value matrix are obtained through memory, and meanwhile, the memory of each layer is ensured to be the same in the reasoning process of a current token, which is equivalent to sharing of a key tensor and a value tensor, so that the effect of cache reduction is realized, and therefore, the computing resource is saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer language processing, in particular to a language model using method of shared memory matrix. BACKGROUND

[0002] In the field of natural language processing, the rise of large language models (LLMs) has witnessed significant technological progress in AI. However, the scale and complexity of these models also pose challenges to inference efficiency. To improve this efficiency, key-value cache (KV Cache) technology has emerged and been widely applied in practice. Key-value cache (KV Cache) is an efficient data storage mechanism that optimizes access speed by caching key-value pair data. In the inference process of language models, certain data may be frequently accessed. KV Cache stores these repeatedly accessed data in memory, enabling fast data retrieval and significantly speeding up inference. It is worth noting that KV Cache is mainly applied in the decoding stage of models, such as in "only decoding" models (e.g. GPT-3, LLaMa3, etc.) and the decoding process of "encoding-decoding" models (e.g. T5). In the inference process of language models, the model generates a corresponding answer based on the input question. This generation process is carried out step by step, with only one token generated each time. The newly generated token is immediately added to the original question to form a new input sequence, which is then passed to the model for further inference. This process is repeated until the model generates a terminator, indicating the completion of the answer. However, in the inference process of large models, as the text length increases and the embedding dimension increases, the KV Cache memory usage also increases rapidly, causing computational power consumption. Based on this, the industry has proposed MQA (Multi Query Attention) and MHA (Grouped Query Attention), which mainly use parameter sharing for each head of multi-head attention. Although the memory usage problem has been alleviated, the effect is still limited. SUMMARY

[0003] The present patent aims to propose a language model structure with shared memory matrix, aiming to reduce the problem of excessive cache usage in the inference process of long text by KV Cache. At the same time, the present patent belongs to a small text generation language model.

[0004] To solve the problems of the prior art, the following technical solutions are adopted. A language model using method of shared memory matrix includes the following steps:

[0005] S1: Make tokens and define special symbols

[0006] Collect open source language training set, use SentencePiece tool for training, including reserved language model [CLS] text start symbol, [PAD] text filler, agreed max_len=2048 input model text maximum length, also the maximum length of model output;

[0007] S2: Construct model:

[0008] The current training text is recorded as {Text}, which is divided into a word list using the tokenize segmentation tool, and [CLS] is added at the beginning. The tokens list without [CLS] at the beginning is the actual model output, that is, the input [CLS] needs to be predicted as the first token of the serialized {Text} sequence at the output end. The memory vector is used as a history storage, initialized as None, and the [CLS] embedding vector is used as the query role. The memory is processed by two fully connected tensors to obtain key and value tensors, which are then cross-attention calculated with the query. Then, the post-feedforward neural network is used to obtain the current token's first layer embedding vector, which is used as the second layer query tensor representation. However, the key and value tensors obtained by the full connection of the memory tensor in each subsequent layer are the same as the first layer representation when training or reasoning the current token. That is, the key and value tensors of each subsequent layer are obtained using the same memory tensor. The first cycle memory only stores the [CLS] embedding vector, which is then processed by two fully connected tensors to obtain key and value tensors. The first [CLS] vector plays the role of query tensor, and cross-attention is calculated. The second cycle memory appends the [CLS] embedding vector and the second input token vector, which is then processed by two fully connected tensors to obtain key and value tensors. The first input token vector plays the role of query tensor and can be processed by multiple layers. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, which is then processed by two fully connected tensors to obtain key and value tensors. The process is repeated for each subsequent cycle.

[0009] S3: Inference stage:

[0010] After tokenizing the [CLS] and the user's input content, splice them into memory, the last token of the user's input content plays the role of the query tensor, calculate the cross-attention, feedforward layer until the output, get the vector representation of the first token, then decode through the full connection to get the generated first token, then splice the token through the embedding layer to the memory tensor, at the same time the token plays the role of the query tensor, repeat the above operation, that is, the final model text generation content can be completed.

[0011] The beneficial effects of the present application are to solve the problem of excessive memory occupation of KV Cache in the reasoning process of ultra-long text in the industry. The present application uses a cyclic iteration structure to obtain the Key matrix and the Value matrix through memory, while ensuring that the memory of each layer is the same during the reasoning of the current token, which is equivalent to sharing the key tensor and the value tensor, thereby achieving the effect of reducing the cache, and the computing resource is saved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 The network model flowchart of the present application is used. DETAILED DESCRIPTION

[0013] S1: making tokens and defining special symbols: S1 is an existing technology in the industry, but in order to clearly explain the innovation points of the present application, an open source language training set is collected, and the SentencePiece tool is used for training, which includes the reserved language model [CLS] text start symbol and [PAD] text padding symbol. The present application stipulates that max_len=2048 is the maximum length of the text input into the model, which is also the maximum length of the model output. However, the user of the present application can train a longer model according to his own situation.

[0014] S2: constructing a model: this is the most important innovation point of the present application, and the model structure is shown in Figure 1 The present application is consistent with the principle of Llama and GPT language models in the reasoning stage in terms of technical implementation. In the main large language model reasoning process, the use of KV Cache technology for ultra-long text reasoning will gradually increase the Keys and Values cache, occupying a large amount of video memory. Multi-Query Attention and Group-Query Attention achieve resource reduction through parameter sharing, but this solution is far from enough for ultra-long text. The model proposed by the present application is as follows Figure 1As shown. The patent will change the training stage of Llama type to a cyclic serial mode, aiming to avoid the problem that the cache content is difficult to optimize due to the causal attention matrix in the current large language model. The large language model structure proposed in the patent is referred to Figure 1, the current training text is denoted as {Text}, and the tokenize segmentation tool is used to divide it into a word list, and a [CLS] symbol is added at the beginning to form the tokens list for model input. The tokens list without [CLS] at the beginning is the actual model output, that is, the input [CLS] needs to be predicted as the first token of the output {Text} after serialization. The memory vector is used as a history storage, and is initialized as None. The [CLS] embedding vector is used as the query role. The memory is processed by two fully connected layers to obtain the key tensor and the value tensor, and then cross-attention calculation is performed with the query. At the same time, the first layer embedding vector of the current token is obtained after the front feedforward neural network, which is used as the second layer query new tensor representation. However, the key tensor and the value tensor obtained by the full connection of the memory tensor of each subsequent layer are the same as the first layer representation during the training or inference of the current token. That is, the key tensor and the value tensor of each subsequent layer are obtained by sharing the same memory tensor, which is very obvious for large language models. If a large language model such as GPT has N layers, the memory occupation of the present patent will be reduced by 1 / N without introducing other methods. Therefore, the effect of using the present patent for large language models is very obvious. The semantic understanding ability of the query tensor of the present patent increases with the increase of the number of model layers, which improves the expression ability of the model. The first cycle memory only stores the [CLS] embedding vector, and then two fully connected layers are used to obtain the key tensor and the value tensor, respectively. The first [CLS] vector plays the role of query tensor, and then cross-attention is calculated. Until multiple layers, the second cycle memory appends the [CLS] embedding vector and the first input token vector, and then two fully connected layers are used to obtain the key tensor and the value tensor, respectively. The first input token vector plays the role of query tensor, and can be processed by multiple layers. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two fully connected layers are used to obtain the key tensor and the value tensor, respectively. In the process of generating long text, the present patent avoids the problem of gradually increasing the size of the KV Cache cache content with the increase of the number of layers by using the memory tensor cache sharing method, that is, the computing resources are saved. The above content is the main protection point of the present patent.

[0015] S3: Inference phase: In the inference phase, the [CLS] and user input are tokenized and concatenated into memory. The last token of the user input plays the role of the query tensor. Cross-attention and feedforward layers are calculated until the output is obtained. The first token's vector representation is then obtained through full connection and decoding to generate the first token. The token is then concatenated to the memory tensor through the embedding layer, and the token plays the role of the query tensor. Repeat the above operations to complete the final model text generation content.

[0016] Noun explanation

[0017] Shared memory matrix: The shared memory matrix mainly remembers the input large language model history token embedding tensor through the cache method. For example, the model output of predicting the 3rd position of the language model, the memory matrix needs to cache the embedding vectors of the model start symbol, the first actual token of the model, and the second actual token of the model. At the same time, the embedding vector of the second actual token of the model input is calculated with the three token embedding vectors in the memory matrix to obtain cross-attention. The memory matrix Memory is equivalent to playing the role of key tensor and value tensor in Transformer.

[0018] KV Cache: KV Cache is an efficient inference acceleration technology specifically designed for large language models. It retains and appends the key and value obtained by historical calculation at each stage of text generation to build a rich parameter cache library. This cache library allows the model to reuse previous calculation results, effectively reducing redundant operations in text generation tasks and significantly improving the efficiency of the model.

[0019] Language model: A language model is a statistical model whose main goal is to predict the probability of a given sequence (usually a text sequence). In the field of natural language processing (NLP), language models are key components and are widely used in machine translation, speech recognition, automatic summarization, sentiment analysis, and dialogue systems.

[0020] Transformer: Transformer language model is a text translation language model based on multi-head self-attention mechanism. In the task of machine translation, it surpasses previous models that rely on recurrent neural networks (RNN).

[0021] token: In language models, a token is usually the smallest unit of text, often a single-character token, a word token, a subword token, a punctuation mark, a number, and so on. Tokens are used to represent a portion of the model input, and each token has a corresponding number. In the embedding layer of a language model, a vector representation is usually assigned to each token. Tokens have a broader meaning nowadays, and a token can be a tool name or even an activity name that people abstractly define.

Claims

1. A language model use method of sharing a memory matrix, characterized by Comprising the following steps: S1: Make token, define special symbol: Collect open source language training set, use SentencePiece tool for training, which includes reserved language model [CLS] text start symbol, [PAD] text padding symbol, agreement max_len=2048 input model text maximum length, which is also the maximum length of model output; S2: Construct model: The current training text is recorded as {Text}, which is divided into a word list using the tokenize segmentation tool, and [CLS] is added at the beginning. The tokens list without [CLS] at the beginning is the actual model output, that is, the input [CLS] needs to be predicted as the first token of the serialized {Text} sequence at the output end; the memory vector is used as a history storage, initialized as None, and the [CLS] embedding vector is used as the query role. The memory is obtained by two full connections to get the key tensor and value tensor, and then cross attention calculation is performed with the query. After that, the front feed neural network is used to get the current token first layer embedding vector new representation, which is used as the second layer query new tensor representation. However, the key tensor and value tensor obtained by full connection of the memory tensor of each subsequent layer are the same as the first layer representation when training or reasoning the current token. That is, the key tensor and value tensor of each subsequent layer are obtained by full connection of the same memory tensor. The first cycle memory only stores the [CLS] embedding vector, and then two full connections are performed to get the key and value tensors. The first [CLS] vector plays the role of query tensor, and then cross attention is calculated. After multiple layers, the second cycle memory appends the [CLS] embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The first input token vector plays the role of query tensor, and can be processed through multiple layers. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle memory appends the second cycle memory embedding vector and the second input token vector, and then two full connections are performed to get the key and value tensors. The third cycle ​ After tokenizing the [CLS] and the user's input content, concatenate them as memory, the last token of the user's input content plays the role of the query tensor, calculate the cross-attention, the feedforward layer until the output, get the vector representation of the first token, then decode through the full connection to get the generated first token, then concatenate the token through the embedding layer to the memory tensor, at the same time, the token plays the role of the query tensor, repeat the above operation, and the final model text generation content can be completed.

Images