A personalized code generation system and method based on residual learning

Abstract

The application discloses a personalized code generation system and method based on residual learning. 1) Collecting code snippet data sets of programming languages, collecting style sequences corresponding to the code snippets through a code detection tool; 2) Setting residual terms based on the code of step 1) and the corresponding style sequence, and generating training data for residual learning; 3) Setting a learnable vector representation for each code style specification; 4) Building a residual learning framework through an LLM; 5) Obtaining the vector representation of each code style specification through residual learning of the residual learning framework of step 4); 6) Obtaining the style sequence of the code to be generated through a code detection tool, using the trained vector representation of the code style specification, and generating personalized code consistent with the historical code style. The application combines the code features learned through residual learning to form the code style of the developer, and guides the large language model to generate personalized code through the code style features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing, and specifically refers to a personalized code generation system and method based on residual learning. BACKGROUND

[0002] With the increasing complexity of software development, automatic code generation technology has become one of the key means to improve development efficiency and code quality. Traditional code generation methods, such as rule-based methods and simple machine learning models, often have difficulty capturing subtle differences and personalized needs in coding.

[0003] In recent years, deep learning technology, especially large pre-trained language models (such as GPT-3, Codex, etc.), has shown great potential in the field of code generation. These models can learn the syntax and structure of programming languages based on massive code libraries, enabling automatic code completion, bug fixing, code refactoring, and other functions. However, although these models perform well in understanding and generating code, they often lack the ability to understand and adapt to developers' personalized coding styles and preferences. Code that has not been personalized often needs to be reviewed and modified by developers, and is very easy to miss, greatly increasing the cost of code development and maintenance.

[0004] Although large language models have strong language understanding and generation capabilities in the field of code generation, how to generate personalized code is still a challenge. Traditional personalized methods cannot fully understand the code style habits of developers and meet the needs of personalized generation, so a learning method that can capture the syntax style of developers' code is needed to meet the personalized needs of different developers.

[0005] Given the excellent code generation effect and performance of deep pre-training models, we introduce them into the personalized generation task. In order to better achieve personalized code generation, we need to find a personalized input and output method that fits the characteristics of large language models, effectively capture and imitate the code style and preferences of developers, and effectively establish fine-grained code style features for different developers. SUMMARY

[0006] To solve the problems in the background art, the present application proposes a novel residual learning framework, based on which a personalized code generation method based on residual learning is provided. The purpose of the present application is to design a method for representing the code style of developers, decompose the code style of users into different code features, combine the code features learned by residual learning to form the code style of developers, and guide the large language model to generate corresponding personalized code through code style features.

[0007] The technical solution adopted by the present application is as follows, specifically comprising the following steps:

[0008] One, a personalized code generation system based on residual learning, comprising:

[0009] A pre-processing module, which performs style detection on the user's historical code to obtain a style sequence corresponding to the code snippet;

[0010] An information extraction module, which constructs a residual learning framework based on a large language model (LLM) based on the user's historical code and the style sequence, and extracts the user's style information through the residual learning framework; the style information is a vector representation of each code style specification learned by training;

[0011] A code generation module that fuses the input natural language and style information to complete the code generation task with the help of a large language model (LLM).

[0012] Two, a personalized code generation method based on residual learning

[0013] Applied to the above personalized code generation system, the personalized code generation method specifically includes the following steps:

[0014] 1) Define several code style specifications from the perspectives of naming, format, and structure, obtain a code snippet dataset of a programming language, and use a code detection tool to collect a style sequence representing the corresponding code snippet;

[0015] 2) Set a residual term based on the code and its corresponding style sequence of step 1), and generate training data that can be used for residual learning;

[0016] 3) Set a learnable vector representation for each code style specification;

[0017] 4) Based on the learnable vector representation of step 3), construct a residual learning framework through a large language model (LLM);

[0018] 5) Based on the training data of step 2), perform residual learning through the residual learning framework of step 4), and train to obtain a vector representation of each code style specification;

[0019] 6) For a developer's historical code, obtain its style sequence through a code detection tool, and use the vector representation of the code style specification trained in step 5) to generate personalized code consistent with the developer's historical code style.

[0020] The step 1) is specifically:

[0021] Each code snippet C in the collected code snippet dataset of the programming language is collected through a code detection tool from the perspectives of naming, format, and structure to obtain a style sequence S = {S1, S2, …, S n};

[0022] wherein the style sequence is a sequence composed of code style specifications as sequence elements; n is the number of collected code style specifications, and the element S n takes value 0 or 1, representing whether to follow the code style specification at the current position, 1 representing following, and 0 representing not following.

[0023] The code style specification includes: naming specification such as adopting camel case for variable naming; format specification such as limiting line length to no more than 50 characters, using spaces on both sides of an operator, etc.; structure specification such as following the order of public method first and private method second in a class, etc.

[0024] The step 2) is specifically:

[0025] The code snippet in step 1) and its corresponding style sequence are traversed to construct a residual learning data set D satisfying the following conditions:

[0026] Each residual learning data d in the residual learning data set D is: A , S A , C B , S B , r, t, T};

[0027] wherein S A , S B are two sequences corresponding to the code snippet C A and the code snippet C B ; elements with value 1 in S B are extracted and composed into a sequence S B '; elements with value 1 in S A are extracted and composed into a sequence S A '; the code snippet C A and the code snippet C B need to satisfy that the number of different elements in the corresponding two sequences S A ', S B ' is 1.

[0028] wherein r represents the code style specification name of the different elements in the two sequences S A ', S B '; t is the text detailed explanation of the code style specification of r.

[0029] For each residual learning data d, the training text T is expressed as: "Given two pieces of code and their corresponding style sequences; please identify and explain the code style specification name appearing in the second piece of code but not appearing in the first piece of code; the first piece of code: C A , the style sequence of the first piece of code: S A ; the second piece of code: C BThe style sequence of the second code segment: S B Answer: 'r' exists in the second code segment but not in the first; its literal interpretation is 't'.

[0030] Step 3) specifically includes the following steps:

[0031] Based on n defined code style guidelines, a learnable vector representation is assigned to each code style guideline, resulting in a set of learnable vector representations A = {A1, A2, ..., A...}. n}

[0032] Step 4) involves constructing the residual learning framework model, specifically including the following steps:

[0033] 4.1) For each training text T corresponding to a residual learning data: insert the corresponding learnable vector representation at the position where the style sequence value is 1, delete the position where the value is 0 from the text, replace r with the corresponding learnable vector representation (find the element position of r in the style sequence, and replace r with the corresponding learnable vector representation according to the element position), and transform the remaining positions into text embeddings through the LLM Tokenizer to finally obtain the training embedding Q.

[0034] 4.2) Freeze all the parameter weights of the LLM, input the training embedding Q into the LLM, and retain the weight learning of the learnable vector representation.

[0035] Step 5) specifically includes the following steps:

[0036] The training data from step 2) is trained using the residual learning framework from step 4) to learn the vector representation of each code style specification; the negative log-likehood function is selected as the final loss function.

[0037]

[0038] Error backpropagation is used to adjust the parameters of the learnable vector representation; the parameters of the learnable vector representation are selected based on the loss function of the model during training, and the set of parameters corresponding to the minimum loss function is used as the parameters of the learnable vector representation.

[0039] Step 6) specifically includes the following steps:

[0040] 6.1) For a developer's historical code, use code inspection tools to collect a style sequence S = {S1, S2, ..., S...} representing the corresponding code segments from three perspectives: naming, formatting, and structure. n};

[0041] 6.2) Represent the style specifications with a value of 1 in the style sequence by forming a vector group A = {A1, A2, ..., A...} to represent the developer's code style characteristics. m}; where m is the number of elements with a value of 1 in the style sequence;

[0042] 6.3) The question text, after being processed by the LLM tokenizer, yields an embedding sequence E = {E1, E2, ..., E...} n}, where n is the number of tokenized vector representations; the question text is a user programming question or requirement described in natural language;

[0043] The style feature vector set A is concatenated with the embedding sequence E to obtain the user's input embedding Q:

[0044] Q = {A1, A2, ..., A} m E1, E2, ..., E n};

[0045] 6.4) The embedded Q input LLM is decoded by autoregression to obtain personalized code consistent with the developer's historical style.

[0046] III. A type of terminal

[0047] The terminal includes a memory, a processor, and a personalized code generation program stored in the memory and capable of running on the processor. When the processor executes the personalized code generation program, it implements the aforementioned personalized code generation method based on residual learning.

[0048] IV. A computer-readable storage medium

[0049] The computer-readable storage medium stores a personalized code generation program, which, when executed by a processor, implements the aforementioned personalized code generation method based on residual learning.

[0050] The beneficial effects of this invention are:

[0051] This invention provides a personalized code generation system and method based on residual learning. It extracts style specifications from data and designs residual learning to train different style specification representations. These style specification representations allow for flexible and diverse code style configurations, helping developers generate personalized code consistent with style requirements. This improves the customization level of automated code generation, increases development efficiency, and reduces code maintenance costs. Attached Figure Description

[0052] Figure 1 This is a flowchart of the present invention;

[0053] Figure 2Personalized code example diagrams are generated for this invention;

[0054] Figure 3 This is a schematic diagram of the basic structure of the personalized code generation method based on residual learning in this invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] like Figure 1 As shown, a personalized code generation method based on residual learning according to the present invention includes the following steps:

[0057] The technical solution adopted in this invention is as follows, specifically including the following steps:

[0058] Step 1): Collect a dataset of Java code snippets. Use code analysis tools to collect several code style guidelines from three perspectives: naming, formatting, and structure, to obtain the style sequence of the corresponding code snippets.

[0059] For each code snippet C in the collected Java code snippet dataset, a code inspection tool is used to collect several code style specification information from three perspectives: naming, formatting, and structure, resulting in a style sequence S = {S1, S2, ..., S...} for the corresponding code snippet. n}, where n is the number of code style guidelines collected. The elements in the style sequence take the value 0 or 1, representing whether a specific code style guideline is followed, with 1 representing "yes" and 0 representing "no".

[0060] like Figure 1 As shown in the "Style Sequence Corresponding to Code", the style sequence corresponding to code A is {1, 1, 0........, 0}; the style specification meaning of the first position is "whether ( and { are on the same line", and the style specification meaning of the second position is "whether camelCase is used". A value of 1 in the style sequence indicates that the corresponding style specification is "yes", and a value of 0 indicates that the corresponding style specification is "no".

[0061] Step 2: Based on the code from Step 1) and its corresponding style sequence, training data suitable for residual learning is generated by setting residual terms.

[0062] For the code in step 1) and its corresponding style sequence, construct a residual learning dataset D that satisfies the following conditions: for each residual learning data d = {C} A S A C B S B r, t, T}:

[0063] Extract S BThe elements with a median value of 1 are arranged into a sequence S. B ′;Extract S A The elements with a median value of 1 are arranged into a sequence S. A ′, where code snippet C A With code snippet C B The two corresponding sequences S′ must satisfy the condition. A S B The number of distinct elements in ' is 1;

[0064] Where r represents two sequences S′ A S B The code style specification names of the distinct elements in 'r', and t is the textual explanation of the code style specification for r. For each residual learning data d, we construct the following training text T: "Given two code snippets and their corresponding style sequences. Please identify and explain the code style specification names that appear in the second code snippet but not in the first code snippet. First code snippet: C..." A The style sequence of the first code snippet: S A Second code snippet: C B The style sequence of the second code segment: S B Answer: 'r' exists in the second code segment but not in the first; its literal interpretation is 't'.

[0065] like Figure 3 As shown, two similar codes and their corresponding style sequences, along with text prompts, form the training text.

[0066] Step 3): Set a learnable vector representation for each code style specification;

[0067] Based on the collected n code style guidelines, a learnable vector representation is set for each code style guideline, resulting in a set of learnable vector representations A = {A1, A2, ..., A...}. n};

[0068] like Figure 3 As shown, the learnable vectors after style sequence transformation of the first code are represented as {a1, a2, a3}; the learnable vectors after style sequence transformation of the second code are represented as {a1, a2, a3, a4}.

[0069] Step 4): Based on the learnable vector representation from Step 3), construct a learning framework for residual learning using the Large Language Model (LLM).

[0070] Step (4) specifically involves:

[0071] 4.1) For a training text T of residual learning data, insert the corresponding learnable vector representation at the position where the style sequence value is 1, delete the position where the value is 0 from the text, and replace r with the corresponding learnable vector representation. The remaining positions are transformed into text embeddings through the LLM Tokenizer, and finally the training embedding Q is obtained.

[0072] 4.2) Freeze all the parameter weights of the LLM, input the training embedding Q into the LLM, and retain the weight learning of the learnable vector representation;

[0073] like Figure 3 As shown, a frozen CodeLlama model is used as the LLM in the residual learning framework. The text is converted into text embeddings through CodeLlama's Tokenizer. At the same time, the style sequence is converted into corresponding learnable vector sets {a1, a2, a3} and {a1, a2, a3, a4}.

[0074] Step 5): Based on the training data from Step 2), train the code using the residual learning framework from Step 4 to learn the vector representation of each code style specification. The negative log-likehood function is selected as the final loss function.

[0075]

[0076] Error backpropagation is used to adjust the parameters of the learnable vector representation; the parameters of the learnable vector representation are selected based on the loss function of the model during training, and the set of parameters corresponding to the minimum loss function is used as the parameters of the learnable vector representation.

[0077] like Figure 1 As shown in the “Training Process”, we train the model based on the residual learning framework using training data.

[0078] Step 6): Based on a developer's historical code, obtain its style sequence through a code inspection tool, and use the vector representation of the code style specification trained in Step 5) to generate personalized code that is consistent with the developer's historical code style;

[0079] Step (6) specifically involves:

[0080] 6.1) For a developer's historical code, we use a code inspection tool to collect several code style guidelines from three perspectives: naming, formatting, and structure. This represents the style sequence guidelines for the corresponding code segments, resulting in a style sequence S = {S1, S2, ..., S...} representing the corresponding code segments. n}, where n is the number of code style guidelines collected. The elements in the style sequence take the value 0 or 1, representing whether a specific code style guideline is followed, with 1 representing "yes" and 0 representing "no".

[0081] like Figure 1 As shown in the "Reasoning Flow" section, if an element of the summarized user style sequence has a value of 1 in the style sequence corresponding to the historical code, then it is assigned a value of 1; for style specifications that have always been assigned a value of 0 in the style sequence of the historical code, their value is 0.

[0082] 6.2) Represent the style specifications with a value of 1 in the style sequence by forming a vector group A = {A1, A2, ..., A...} to represent the developer's code style characteristics. m}; where m equals the number of elements with a value of 1 in the developer's style sequence.

[0083] 6.3) Combine the vector group A of style features with the vector sequence E = {E1, E2, ..., E...} of the question text after using an LLM tokenizer. n} are concatenated to obtain the user's input embedding Q = {A1, A2, ..., A} m E1, E2, ..., E n}, where n is the number of tokenized vector representations.

[0084] The embedded Q input LLM is decoded via autoregression to obtain personalized code consistent with the developer's historical style.

[0085] like Figure 1 As shown in the "Inference Flow" section, the input problem is "Given an array of integers, write a function to calculate the prefix sum of the array," and the user-style sequence is {1, 1, 0, ..., 0}. The problem and user-style sequence are input into a pre-trained model based on a residual learning framework, ultimately producing code that conforms to the user's coding style. A code example is shown below. Figure 2 As shown, the generated code has "{" and "(" on the same line, and the class name "PrefixSum", variable name "prefixSums", and "calculatePrefixSums" are all camelCase, which conforms to the user's historical style habits.

[0086] The results of this method on the personalized code dataset after it has been organized are as follows:

[0087] Personalization method BLEU-4 Rouge-1 Rouge-2 DAPT 0.3078 0.4001 0.2877 Prefix-Tuning 0.4835 0.3623 0.2511 Inventive model 0.5573 0.4277 0.2917

[0088] Note:

[0089] (1) DAPT and Prefix-Tuning are commonly used personalized algorithm models;

[0090] (2)BLEU-4:BLEU-4∈[0,1], the larger the value, the better the personalization effect;

[0091] (3) Rouge-1: NMI∈[0,1], the larger the value, the better the personalization effect;

[0092] (4) Rouge-2: FMI∈[0,1], the larger the value, the better the personalization effect;

[0093] (5) Compared with DAPT, the metrics BLEU-4, Rouge-1, and Rouge-2 on the same dataset in this invention are improved by 24.95%, 2.66%, and 0.4%, respectively;

[0094] (6) Compared with Prefix-Tuning, the metrics BLEU-4, Rouge-1, and Rouge-2 on the same dataset in this invention are improved by 7.38%, 6.54%, and 4.06%, respectively.

Claims

1. A personalized code generation system based on residual learning, characterized in that, include: The preprocessing module performs style detection on the user's historical code to obtain the style sequence of the corresponding code snippets; The information extraction module constructs a residual learning framework based on the user's historical code and style sequence, using a large language model (LLM) to extract the user's style information; the style information is a vector representation of each code style specification learned during training. The code generation module fuses the input natural language and style information, and uses a large language model (LLM) to complete the code generation task. The style sequence of the code snippets is S = {S1, S2, ..., S...} n The code style specifications were collected from three perspectives: naming, formatting, and structure, using code analysis tools. The style sequence is a sequence composed of code style guidelines as its elements; n represents the number of collected code style guidelines, and S represents the number of elements in the style sequence. n The value is 0 or 1, which indicates whether the code style guidelines for the current location are followed. 1 means follow, and 0 means do not follow.

2. A personalized code generation method based on residual learning, characterized in that, The personalized code generation method applied to the personalized code generation system of claim 1 specifically includes the following steps: 1) Define several code style specifications from the perspectives of naming, format, and structure; obtain a dataset of code snippets for programming languages; and use code detection tools to collect style sequences representing the corresponding code snippets. 2) Based on the code in step 1) and its corresponding style sequence, set residual terms to generate training data that can be used for residual learning; 3) Set a learnable vector representation for each code style specification; 4) Based on the learnable vector representation from step 3), a residual learning framework is constructed using the Large Language Model (LLM). 5) Based on the training data from step 2), residual learning is performed using the residual learning framework from step 4) to train and learn the vector representation of each code style specification. 6) For a developer's historical code, obtain its style sequence through code inspection tools, and use the vector representation of the code style specification trained in step 5) to generate personalized code that is consistent with the developer's historical code style.

3. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 1) specifically refers to: For each code snippet C in the collected dataset of programming language code snippets, a style sequence S = {S1, S2, ..., S...} is obtained from the three perspectives of naming, formatting, and structure using a code detection tool. n }; Here, the style sequence is a sequence composed of code style specifications as sequence elements; n is the number of collected code style specifications, and S is the element in the style sequence. n The value is 0 or 1, which indicates whether the code style guidelines for the current location are followed. 1 means follow, and 0 means do not follow.

4. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 2) specifically refers to: Iterate through the code snippets in step 1) and their corresponding style sequences to construct a residual learning dataset D that satisfies the following conditions: Each residual learning data point d = {C} in the residual learning dataset D A ,S A C B ,S B ,r,t,T}; Among them, S A ,S B For code snippet C A With code snippet C B The two corresponding sequences; extract S B The elements with a median value of 1 are arranged into a sequence S. B ′;Extract S A The elements with a median value of 1 are arranged into a sequence S. A ′;Code snippet C A With code snippet C B The two corresponding sequences S must satisfy the condition. A ′ ,S B The number of distinct elements in ' is 1; Where r represents two sequences S A ′,S B The code style specification name for different elements in 'r'; t is a detailed textual explanation of the code style specification for r; For each residual learning data d, the training text T is represented as: "Given two code snippets and their corresponding style sequences; please identify and explain the code style specification names that appear in the second code snippet but not in the first code snippet; First code snippet: C..." A The style sequence of the first code snippet: S A Second code snippet: C B The style sequence of the second code segment: S B Answer: 'r' exists in the second code segment but not in the first; its textual interpretation is 't'.

5. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 3) specifically includes the following steps: Based on n defined code style guidelines, a learnable vector representation is assigned to each code style guideline, resulting in a set of learnable vector representations A = {A1, A2, ..., A...}. n } 6. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 4) involves constructing the residual learning framework model, specifically including the following steps: 4.1) For each training text T corresponding to a residual learning data: insert the corresponding learnable vector representation at the position where the style sequence value is 1, delete the position where the value is 0 from the text, replace r with the corresponding learnable vector representation, and transform the remaining positions into text embeddings through the LLM Tokenizer to finally obtain the training embedding Q; 4.2) Freeze all the parameter weights of the LLM, input the training embedding Q into the LLM, and retain the weight learning of the learnable vector representation.

7. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 5) specifically includes the following steps: The training data from step 2) is trained using the residual learning framework from step 4) to learn the vector representation of each code style specification; the negative log-likehood function is selected as the final loss function. Error backpropagation is used to adjust the parameters of the learnable vector representation; the parameters of the learnable vector representation are selected based on the loss function of the model during training, and the set of parameters corresponding to the minimum loss function is used as the parameters of the learnable vector representation.

8. The personalized code generation method based on residual learning according to claim 2, characterized in that, Step 6) specifically includes the following steps: 6.1) For a developer's historical code, use code inspection tools to collect a style sequence S = {S1, S2, ..., S...} representing the corresponding code segments from three perspectives: naming, formatting, and structure. n }; 6.2) Represent the style specifications with a value of 1 in the style sequence by forming a vector group A = {A1, A2, ..., A...} to represent the developer's code style characteristics. m }; where m is the number of elements with a value of 1 in the style sequence; 6.3) The question text, after being processed by the LLM tokenizer, yields an embedding sequence E = {E1, E2, ..., E...} n }, where n is the number of tokenized vector representations; the question text is a user programming question or requirement described in natural language; The style feature vector set A is concatenated with the embedding sequence E to obtain the user's input embedding Q: Q={A1,A2,…,A m E1, E2,..., E n }; 6.4) The embedded Q input LLM is decoded by autoregression to obtain personalized code consistent with the developer's historical style.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a personalized code generation program stored in the memory and executable on the processor. When the processor executes the personalized code generation program, it implements the personalized code generation method based on residual learning as described in any one of claims 2-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a personalized code generation program, which, when executed by a processor, implements the personalized code generation method based on residual learning as described in any one of claims 2-8.

Images