Personal python code automatic completion method

Abstract

The invention relates to the technical field of artificial intelligence data processing, in particular to a personal python code automatic completion method. The method aims to keep the reasoning stage and the training stage consistent, although the training efficiency is reduced, the interaction mode of the current token and the historical token can be controlled more flexibly, and in order to embody the actual value of the mode, the method achieves the purpose that the token far away from the current to-be-predicted position is represented by a tensor in a GRU mode in a unified mode. According to the method, the token which is closer to the current to-be-predicted position is kept unchanged, the mode that the current input token, the ultra-long distance token and the closer token can be subjected to self-attention interaction when the text ultra-long memory variable is fixed is achieved, the cache content is not increased, although the efficiency of the training stage is reduced, the cache content is reduced in the reasoning stage, the calculation amount is reduced, and the training efficiency is improved. And meanwhile, interaction with a long-distance text token is realized, so that the method is more valuable.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence data processing, and in particular to a method for automatically completing a personal Python code. Background Art

[0002] Personal Python code completion refers to the function of providing programmers with customized code completion based on their personal coding habits, coding style and other factors. This personalized code completion tool can provide programmers with more accurate and efficient code completion suggestions based on their personal coding history, project code base and other information, thereby improving coding efficiency and quality. At present, the industry generally uses open source large language models for code completion. However, the code completion capabilities of open source large language models are usually widely used, and many personal unique or innovative codes are not made public. At the same time, open source large language model code completion tools usually have more than 1 billion model parameters, which consumes huge computing power. In the implementation of large language models, code completion, unit testing, code comments, report writing, etc. are usually very long texts. In the reasoning process of large language models, KV Cache avoids the problem of repeated calculation of key and value tensors of historical tokens, thereby improving the reasoning speed of large language models. It is widely used in large language models in the industry. However, as the token representation dimension of today's large language models gradually increases, the number of layers gradually increases, and the reasoning length is too long, the cache content will occupy a large amount of video memory resources, resulting in model reasoning errors. This patent proposes a text generation (personal python code completion) language model with remote GRU and proximal Attention. Summary of the invention

[0003] Code completion can automatically complete subsequent code snippets based on the partial code entered by the user, saving developers time in manually entering code and improving coding efficiency. However, the open source large language model does not use the code of programmers with personal style, so the actual value of code completion is very limited. Usually, if you want to write code in your own style, you need targeted training. Therefore, the value of open source large language models for personal code completion is limited.

[0004] The present invention solves the deficiencies of the prior art and provides the following technical solutions: S1. Create a personal code token vocabulary and training corpus: Collect personal python codes, make training corpus, segment personal codes into token_personal, and count the word frequency of all personal python codes through reverse dictionary matching word segmentation algorithm. The python codes include reserved [CLS] text start symbol and [PAD] text filler. The [PAD] text filler is used to automatically complete the python personal codes. S2. Model training: The Llama3-like and GPT-like language models are cyclically combined, and GRU processes the ultra-remote token embedding information, that is, all token embeddings far away from the current generated token are represented by an embedding vector, and this embedding vector representation is obtained by the GRU model, while the proximal token interacts with the current token using a causal attention method; the model training stage is as follows: a user's python code is Python_txt, and the tokenize word segmentation tool is used to divide it into a word list, and the [CLS] start symbol is added to the front. The memory vector is used as a historical storage, initially None, and the [CLS] embedding vector. During the training process, the first cycle memory is assigned to the [CLS] embedding vector, and the [CLS] embedding vector is based on the query angle. Color, memory is fully connected through two tensors to play the role of key tensor and value tensor for cross attention calculation, and then passes through the feedforward neural network to obtain the possibility tensor of the token at the output position of the first input token. In the second cycle, memory concatenates the [CLS] embedding vector with the second input token vector, and then passes through two full connections to obtain the tensors key and value respectively, and then concatenates them with the second input token vector to calculate attention and feedforward neural network, and so on after multiple layers; when the memory tensor size is 1000, then at this time, each time a new tensor is concatenated in the memory, the first tensor and the second tensor stored in the memory are calculated using the GRU method and expressed as Embed_start, Although memory adds a new embedding tensor, the first tensor Embed_0 and the second tensor Embed_1 are represented by the Embed_start vector, so as to ensure that the memory tensor is always 1000, that is, the first token stores the information summary of the extremely distant token tensors in history, and the token tensor whose distance to the current token does not exceed 1000 is the token tensor itself; S3, reasoning stage: using GPT and Llama loop implementation, and following the steps of S2 training stage for reasoning stage.

[0005] Beneficial effects of the invention: The invention content to be protected by this patent is: This patent proposes a cyclic training implementation method for GPT-type and Llama-type language models, aiming to keep the inference stage and the training stage consistent. Although the training efficiency is reduced, the interaction method of the current token and the historical token can be more flexibly controlled. In order to reflect the actual value of this method, this patent implements a token that is far away from the current predicted position using a tensor uniform representation using the GRU method, and the token that is closer to the current predicted position remains unchanged, so that when the text ultra-long memory variable is fixed, the current input token and the ultra-long distance token and the closer token can all interact with each other through self-attention, and the cache content does not increase. Although the efficiency of the training stage is reduced, the cache content is reduced in the inference stage, the amount of calculation is reduced, and the interaction with the distant text token is realized, which is more valuable. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 The network model structure diagram of this patent. DETAILED DESCRIPTION

[0007] The following is further described in conjunction with the embodiment: a method for automatically completing personal python code includes the following steps: S1: Create a personal code token vocabulary and training corpus: S1 is a necessary step for training text completion and text generation models. It is a mature technology in the industry and not a technical innovation of this patent. The main purpose here is to express the integrity of the implementation of this patent. Personal python code is collected to create training corpus. For example, a python code is demo.py, with a total of 90 lines of code. This patent stipulates to complete four lines of code, such as starting the loop from the first line of demo.py, [1, 2, 3, 4], [2, 3, 4, 5], ... [87, 88, 89, 90] is a training set, [i, j, k, l] where i is the i-th line of code, j is the j-th line of code, k is the k-th line of code, and l is the l-th line of code. Line breaks in a training set are represented by the \n character, and the Tab key and line break at the beginning of each training set are deleted. The token decoding of the open source large language model is recorded as token_open=70000 (existing technology in the industry); for personal code, use :, ", ', =, ,, >, <, space, line break, \n, and Tab keys to split the personal code into token_personal. For example, a piece of code in the large language model from transformers import Taking AutoModelForCausalLM as an example, the segmented tokens are from, transformers, import, AutoModelForCausalLM, a total of 4 tokens, and token_personal produces 20,000 tokens; SentencePiece training is used for personal code to obtain a total of 10,000 tokens, recorded as token_sp; all tokens of token_open, token_sentencepiece, and token_personal are summarized to a total of 100,000 tokens, and all personal python codes are statistically analyzed for word frequency through a reverse dictionary matching word segmentation algorithm to screen 30,000 high-frequency tokens, and low-frequency tokens are converted into character forms to avoid abnormal symbols during user use. After sorting, a total of 40,000 model training tokens are obtained, including the reserved [CLS] text start character and [PAD] text filler. The automatic completion of python personal code is mainly for quickly completing the user's personal python code. Therefore, this patent stipulates that max_len=2048 is the maximum text length of the input model stipulated in this patent, and it is also the maximum length of the model output. People using this patent can adjust the length to suit themselves as needed.

[0008] S2: Model training: S2 is the main innovation of this patent. The model structure is shown in Figure 1, this patent designs a layer of neural network model. The cache technology solution proposed in this patent can be modified to apply to multiple layers. Just for the convenience of explanation, this patent is designed as a layer here. In the current text generation large language model reasoning process, the use of KV_Cache technology for super-long text reasoning will gradually increase the number of Keys and Values ​​caches, occupying a large amount of computing resources. Although Multi-QueryAttention and Group-Query Attention can reduce resources through parameter sharing, they are far from enough for super-long texts. The model proposed in this patent is Figure 1As shown. This patent transforms the Llama3-like and GPT-like language model implementations into a loop mode, aiming to avoid the causal attention matrix in GPT, so that the text generation language model remains completely consistent during the training and reasoning phase, thereby better controlling the interaction between the current token and the historical token. In order to reflect the technical advantages of this patent, this patent designs a GRU to process ultra-far-end token embedding information, that is, all token embeddings far from the current generated token are all represented by an embedding vector, and this embedding vector representation is obtained by the GRU model, while the proximal token uses the causal attention method to interact with the current token. The model training phase is described as follows: A user's python code is Python_txt, which is divided into a word list using the tokenize word segmentation tool, and the [CLS] start symbol is added at the front. The memory vector is used as the historical storage, and is initially None. The [CLS] embedding vector is used during the training process. In the first loop, the memory is assigned the [CLS] embedding vector. The [CLS] embedding vector plays the role of query. The memory is connected through two full connections to obtain tensors that play the roles of key tensor and value tensor respectively for cross-attention calculation. At the same time, it passes through the feedforward neural network to obtain the possibility tensor of the token at the output position where the first input token is located. In the second loop, the memory concatenates the [CLS] embedding vector with the second input token vector, and then passes through two full connections to obtain the tensors key and value respectively, and then concatenates them with the second input token vector to calculate attention and feedforward neural network. Specifically, it can pass through multiple layers, and so on. When the memory tensor size is 1000, then at this time, each time a new tensor is concatenated in the memory, the first tensor and the second tensor stored in the memory are calculated using the GRU method and expressed as Embed_start, although a new embedding tensor is added to memory, the first tensor Embed_0 and the second tensor Embed_1 are represented by the Embed_start vector, thereby ensuring that the memory tensor is always 1000, that is, the first token stores the information summary of the ultra-distant token tensors in history, and the token tensor whose distance from the current token does not exceed 1000 is the token tensor itself. Through the above method, in the process of generating ultra-long texts, the memory tensor can be guaranteed to remain fixed, avoiding the gradual increase in the amount of calculation and the gradual increase in the cache content. The above content is the main protection point of this patent. The disadvantage of this patent relative to the Llama-like model is that the training phase requires a loop, and the advantage is that the memory cache content can be flexibly controlled through a loop. In order to verify the practicality of this patent, a GPU server with a video memory of 40G is used for training for two days to achieve automatic completion of personal python code.

[0009] S3: Reasoning stage: Because this patent adopts GPT and Llama loop implementation, the training stage and the reasoning stage are exactly the same, and this patent will not be further elaborated.

[0010] The biggest feature of using open source large language models for automatic code completion is the huge consumption of computing power. This patent designs an automatic code completion training method similar to KV_Cache technology as a language model, which can automatically complete the user's personal code. At the same time, this patent is different from KV_Chache in that it uses historical information for sliding window caching. The semantic vector of the remote token (for example, more than 1000 tokens) is uniformly represented as a vector using GRU, and the vectors of tokens with similar distances to the current token (for example, tokens with a distance of no more than 1000) and the calculation methods of GPT-like, Llama-like and other models remain unchanged, aiming to reduce the KV_Cache cache content.

[0011] Glossary Personal Python code auto-completion: Python code auto-completion can help programmers automatically complete subsequent Python codes, assist programmers in programming, and improve programming efficiency. Personal code auto-completion is mainly for personal codes, training personal code auto-completion.

[0012] Language model: A language model is used to understand and generate natural language text. Its main function is to predict the next possible word or character sequence in a given text. The training process of the language model is based on large-scale text data and builds a model by learning the associations between words, phrases, and sentences.

[0013] Text generation language model: The text generation language model predicts and generates new text (new token) based on existing text (or token sequence). It can creatively generate coherent and meaningful new text based on pre-provided text input. In the field of text generation, the most widely used form is the completion form: in this mode, the model receives a partially completed text sequence and predicts the next text content to complete the entire sequence. GRU: The GRU language model is a variant based on a recurrent neural network (RNN). It improves the traditional RNN structure by introducing a gating mechanism to solve the problem of long-term dependency. GRU, like the long short-term memory network (LSTM), is a popular choice for processing sequence data in deep learning. GRU introduces two gating mechanisms - update gate and reset gate, which control the flow and forgetting of information.

[0014] Transformer: Transformer is a deep learning model based on the self-attention mechanism, which is widely used in natural language processing tasks. It performs well in processing long sequence data, and is particularly suitable for tasks such as machine translation that require processing global dependencies. The core of the Transformer model is the attention mechanism, which allows the model to establish connections between different positions, so as to better understand and process the input sequence. Transformer consists of two parts, an encoder and a decoder, each of which consists of multiple layers. In the encoder, the input sequence is processed by multiple layers of self-attention layers and feedforward neural network layers; in the decoder, in addition to the self-attention layers and feedforward neural network layers, it also includes an encoder-decoder attention layer to integrate the information of the encoder with the information of the decoder.

[0015] Feedforward Neural Network (FFN): In the Transformer model, a feedforward neural network is used as part of each layer in the encoder and decoder. In the Transformer, FFN is usually used as a fully connected network structure after the self-attention layer. Specifically, FFN usually consists of two linear layers, each separated by a nonlinear activation function (usually ReLU, GeLU). The dimensions of the two linear layers are usually different, but their input and output dimensions are the same. This structure enables FFN to increase the capacity of the language model by learning nonlinear mappings while keeping the input dimension unchanged.

[0016] Token: In language models, tokens usually refer to the smallest unit of text, usually single-word tokens, word tokens, subword tokens, punctuation marks, numbers, etc. Tokens are used to represent part of the model input. Each token has a corresponding number. In the embedding layer of the language model, each token is usually assigned a vector representation. Today, tokens have a broader meaning. Tokens can be abstract tool names or even activity names that people define.

[0017] KV Cache: KV Cache is a large language model reasoning acceleration method. In the text generation process, the key and value parameters of historical calculations are appended to save the key and value values ​​of each round in the history. The KV Cache variable content stores a lot of historically calculated key and value values, thus avoiding redundant calculations. However, in the process of generating very long texts, it will occupy a lot of video memory resources.

Claims

1. A personal python code automatic completion method, characterized by The steps include: S1. Create a personal code token vocabulary and training corpus: Collect personal python codes, make training corpus, segment personal codes into token_personal, and count the word frequency of all personal python codes through reverse dictionary matching word segmentation algorithm. The python codes include reserved [CLS] text start symbol and [PAD] text filler. The [PAD] text filler is used to automatically complete the python personal codes. S2. Model training: The Llama3-like and GPT-like language models are cyclically combined, and GRU processes the ultra-remote token embedding information, that is, all token embeddings far away from the current generated token are represented by an embedding vector, and this embedding vector representation is obtained by the GRU model, while the proximal token interacts with the current token using a causal attention method; the model training stage is as follows: a user's python code is Python_txt, and the tokenize word segmentation tool is used to divide it into a word list, and the [CLS] start symbol is added to the front. The memory vector is used as a historical storage, initially None, and the [CLS] embedding vector. During the training process, the first cycle memory is assigned to the [CLS] embedding vector, and the [CLS] embedding vector is based on the query angle. Color, memory is fully connected through two tensors to play the role of key tensor and value tensor for cross attention calculation, and then passes through the feedforward neural network to obtain the possibility tensor of the token at the output position of the first input token. In the second cycle, memory concatenates the [CLS] embedding vector with the second input token vector, and then passes through two full connections to obtain the tensors key and value respectively, and then concatenates them with the second input token vector to calculate attention and feedforward neural network, and so on after multiple layers; when the memory tensor size is 1000, then at this time, each time a new tensor is concatenated in the memory, the first tensor and the second tensor stored in the memory are calculated using the GRU method and expressed as Embed_start, Although memory adds a new embedding tensor, the first tensor Embed_0 and the second tensor Embed_1 are represented by the Embed_start vector, so as to ensure that the memory tensor is always 1000, that is, the first token stores the information summary of the extremely distant token tensors in history, and the token tensor whose distance to the current token does not exceed 1000 is the token tensor itself; S3, reasoning stage: using GPT and Llama loop implementation, and following the steps of S2 training stage for reasoning stage.

Images