A method and system for automatic code evolution and editing

Abstract

The application discloses an automatic code evolution and editing method and system, and belongs to the information technology field.The method comprises the following steps: constructing a training data set, wherein the training data set is composed of a plurality of old and new code pairs; generating a loss based on a code editing target, a code preservation segment mask loss, and a code editing loss combined with denoising self-encoding, training a code evolution and editing model on the training set; wherein the code evolution and editing model is constructed based on a Transformer Encoder-Decoder framework; and realizing evolution or editing of a test code based on the code evolution and editing model.The application can efficiently generate correct target code in the process of modifying old code.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a method and system for the evolution and editing of automatic code. Background Technology

[0002] Software requires continuous maintenance and updates to enhance functionality, adapt to new requirements, and fix vulnerabilities. During software maintenance, code evolution and editing tasks—such as editing old code to obtain new code—are important research targets, including tasks like code enhancement and automated code review. Code evolution and editing tasks are challenging because traditional deep learning models and code pre-trained models primarily focus on capturing code semantics and do not learn sufficient context regarding code editing and changes, thus failing to demonstrate good code editing performance. To adapt to the characteristics of code evolution and editing tasks, code pre-trained models need to be specifically designed for code editing.

[0003] Existing technologies for code editing tasks mainly fall into two categories: 1) code change representation methods using masked language models; and 2) automatic code change methods using denoised autoencoder models. The first method first uses a diff tool to represent the versions of the code before and after a change as a sequence, using line-level changes as the basic change granularity. Then, a masked language model is applied to this sequence with change information to learn the representation of code changes, such as predicting whether a line will be modified or predicting the content of obscured code. The second method uses statistical information from downstream tasks to add noise to the original code. This noise includes inserting tokens, deleting tokens, and replacing token content, with the probability of introducing these different types of noise based on statistics from similar downstream tasks. The model then uses a denoised autoencoder model to recover the original input from this noisy input, thereby learning how to generate correct code from noisy code.

[0004] The shortcomings of existing technologies are as follows: First, the code change representation method using masked language models is not suitable for code editing tasks, mainly for the following two reasons: 1) Challenges in scenario adaptation: Code change representation tasks mainly focus on whether the changes in code are consistent with the corresponding natural language descriptions, which is inconsistent with the task scenario of editing old code to obtain new code. Therefore, it is difficult to use deep learning models for change representation to generate the desired target code. 2) Challenges in code editing granularity: Code change representation usually uses line-level code to represent code modifications. However, the editing granularity in development may be much finer, such as code token-level modifications. Second, the automatic code change method using denoised autoencoder models faces the challenge of real code editing. This is because introducing noise only reflects the probability of the code being edited as a whole, without considering that different codes may have different editing models in a specific context. Therefore, the data constructed by introducing noise may deviate from the code editing patterns in the actual development process.

[0005] Therefore, existing methods for code evolution and editing primarily rely on pre-trained code models. These models utilize large-scale code corpora to learn general knowledge about code and editing, and then fine-tune and apply this knowledge to specific downstream tasks. Existing methods based on pre-trained code models mainly fall into two categories: code change representation methods using masked language models and automatic code change methods using denoised autoencoder models. However, both types of methods still face challenges related to scenario adaptation, editing granularity, and access to real-world editing data. These challenges make it difficult to efficiently generate correct target code when modifying old code. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides an automatic code evolution and editing method and system that can efficiently generate correct target code during the modification of old code.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An automatic code evolution and editing method, the method comprising:

[0009] Construct a training dataset, which consists of several pairs of new and old code;

[0010] A code evolution and editing model is trained on the training set based on code editing target generation loss, code preservation fragment masking loss, and code editing loss combined with denoising autoencoder; wherein, the code evolution and editing model is built on the Transformer Encoder-Decoder framework;

[0011] This implements the evolution or editing of test code based on a code evolution and editing model.

[0012] Furthermore, the construction of the training dataset includes:

[0013] Projects are selected based on their popularity on GitHub, and the code of the selected projects is used as the old code.

[0014] Extract merged pull requests from these projects, and obtain new code by extracting code editing patches from the pull requests; wherein the difference between the original old code and the original new code does not exceed p lines, and the number of tokens changed from the original old code to the original new code does not exceed q% of the total number of tokens, where p and q are positive integers;

[0015] After filtering the old and new code based on code length, code content, and code changes, the training dataset is obtained.

[0016] Furthermore, the filtering of old code pairs based on code content includes:

[0017] Remove keyword tags associated with GitHub commits;

[0018] Filter out data whose code content does not contain normal code content, and exclude data that consists only of comments and has no code content.

[0019] Furthermore, the code editing target generates a loss. Where t represents the current generation step, and n t Mask represents the t-th token of the generated edited code. ((ETG)) The method of masking is represented, where N represents the maximum length of the generated sequence, and P represents the maximum length of the generated sequence. (θ) θ represents the probability distribution of the target sequence generated by the model, and θ represents the learnable parameters of the model.

[0020] Furthermore, the code preserves fragment mask loss. Where t represents the current generation step, N represents the maximum length of the generated sequence, and P (θ) Let θ represent the probability distribution of the target sequence generated by the model, and let n represent the learnable parameters of the model. t This represents the t-th token in the generated edited code, o ((keep)) This represents a token that remains unchanged before and after code editing, o (change) A token that shows the changes before and after editing.

[0021] Furthermore, the code editing loss combined with denoising autoencoder... Where t represents the current generation step, N represents the maximum length of the generated sequence, and P (θ)Let θ represent the probability distribution of the target sequence generated by the model, and let n represent the learnable parameters of the model. t This represents the t-th token in the generated edited code. This represents the old code after noise was introduced.

[0022] Furthermore, training a code evolution and editing model on the training set based on the code editing target generation loss, the code preservation fragment masking loss, and the code editing loss combined with denoising autoencoder includes:

[0023] The total loss value is obtained by summing the loss values ​​of code editing target generation loss, code preservation fragment masking loss, and code editing loss combined with denoising autoencoder.

[0024] During the training of each batch, the model parameters are adjusted by minimizing the total loss value to obtain the code evolution and editing model.

[0025] Furthermore, the evolution of test code based on the code evolution and editing model includes:

[0026] The test code is formally described as the input representation x = [CLS], o1, o2, ..., o n [SEP]; where [CLS] is the first special marker in the input representation, [SEP] is the second special marker in the input representation, n represents the number of tokens in the code under test, and o n This represents the token of the nth code segment in the tested code.

[0027] The input representation x is fed into the code evolution and editing model to obtain the evolution result of the test code.

[0028] Furthermore, the editing of the code under test based on the code evolution and editing model includes:

[0029] The test code and edit intentions, which contain natural language forms, are formally described as the input representation x = [CLS], w1, w2, ..., w m [SEP],o1,o2,…,o n [SEP]; where [CLS] is the first special marker in the input representation, [SEP] is the second special marker in the input representation, n represents the number of codes in the tested code, and o n This represents the token of the nth code segment in the tested code, where m represents the number of words in the natural language, and w... m This represents the m-th word in a natural language.

[0030] The input representation x is fed into the code evolution and editing model to obtain the encoding result of the test code.

[0031] An automatic code evolution and editing system, the system comprising:

[0032] The training set construction module is used to construct the training dataset, which consists of several pairs of new and old code.

[0033] The training module is used to train a code evolution and editing model on the training set based on the code editing target generation loss, the code preservation fragment masking loss, and the code editing loss combined with denoising autoencoder; wherein the code evolution and editing model is built on the Transformer Encoder-Decoder framework;

[0034] The testing module is used to evolve or edit test code based on a code evolution and editing model.

[0035] The beneficial effects of this invention are as follows:

[0036] This invention, during its design phase, comprehensively reconsidered the problems of code evolution and editing, resulting in a novel perspective for enhancing code evolution and editing capabilities. Utilizing this new perspective, the invention strengthens the mining and utilization of realistic, fine-grained code editing. However, this new perspective also presents new challenges to the automatic evolution and editing of code. Specifically, it requires large-scale, realistic, fine-grained code editing data samples for training, and also demands that the model generate entirely new code editing targets. This necessitates that the deep learning model and system fully understand the context of code editing to generate code editing objects with higher accuracy. To address these new challenges, this invention designs new pre-trained targets to represent and perceive code editing intentions. Attached Figure Description

[0037] Figure 1 It is a flowchart of real-world, fine-grained code editing data collection.

[0038] Figure 2 This is a diagram of the pre-training framework of the present invention. Detailed Implementation

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0040] This invention realizes an automatic code evolution and editing system, which means that: at the input end, the user can input the code that needs to be edited or modified, or additionally input natural language information according to the scenario to express a clear modification intention; at the output end, the system can automatically synthesize the edited program to fix the problems in the code or solve the user's modification intention.

[0041] Specifically, given the importance of code evolution and code editing issues, this invention relates to a system for automatically generating code editing targets. The input and output of this system are described below:

[0042] At the input field, users provide the code to be edited and their editing intent. Users can enter only the code to be edited, or both the code and a description of how it should be edited. If only the code to be edited is entered, the system will automatically edit the existing code and improve its quality; if both the code and the specific editing intent are entered, the system will modify the code according to that intent.

[0043] At the output end, the system of this invention will automatically generate the target code for editing based on the knowledge learned from a large-scale corpus.

[0044] To generate targets for code evolution and editing, this invention: 1. First, collects fine-grained real-world code editing data; 2. Then, designs pre-training techniques to train the code editing model; 3. Finally, evaluates the generation results of the automatic code evolution and editing system. The performance details are described below:

[0045] I. Data Collection from Fine-Grained Real-World Code Editing

[0046] To support a system for automatic code evolution and editing, this invention employs... Figure 1 The steps shown collect a fine-grained dataset of real code changes: 1) First, select a small number of projects based on their popularity on GitHub; 2) Extract merged pull requests from these projects and further extract code editing patches from these pull requests; 3) Design data filtering rules to clean the data.

[0047] Regarding the data source, this invention chooses GitHub because it is one of the world's largest code hosting platforms, boasting a vast number of open-source projects and contributors. GitHub contains a large amount of pull requests and code commits, reflecting real-world code changes. Furthermore, this invention chooses Java as the target language for data collection primarily for the following reasons: First, Java is one of the most popular programming languages ​​and is widely used in software development. It showcases a large amount of code evolution and change. Second, most tasks related to code evolution and analysis involve the Java programming language. Therefore, by focusing on Java, the dataset involved in this invention can be well adapted to various downstream tasks.

[0048] Regarding data quality, this invention focuses on code change data within merged pull requests. Merged pull requests typically represent code changes that have been reviewed and accepted. This data provides more reliable examples of code changes, helping pre-trained code models learn higher-quality code editing and evolution patterns. A pull request may contain one or more commits, and this invention focuses on fine-grained editing operations within each commit. This approach helps the model learn more specific code editing knowledge. Furthermore, this invention selects a small number of popular open-source projects from GitHub. These projects are highly likely to be well-maintained and feature high-quality code editing.

[0049] Regarding data filtering, this invention relates to three code filtering rules: based on code length, based on code content, and based on code changes. Regarding code length, this invention filters out excessively long code snippets because they are difficult to process and the model struggles to learn from. Specifically, this invention filters out code snippets measured in lines where the length before and after the change exceeds 15 lines. Regarding code content, this invention first removes keyword tags related to GitHub commits, such as "@@". Then, it further filters out data whose code content does not contain normal code content and excludes data consisting only of comments without any code content. Furthermore, this invention further filters out duplicate data. Regarding code changes, it first removes data containing only changes in comments. Secondly, this invention specifically considers fine-grained code edits, rather than processing completely different code before and after the edit. Therefore, the data collection process sets the following limitations on the granularity of code edits: limiting the number of changed lines of code to a difference of no more than 5 lines between two versions; and limiting token changes from the old code to the new code, ensuring that they do not exceed 60% of the total number of tokens in the old and new code. This approach helps capture important, fine-grained, and easily understood code changes.

[0050] The open-source projects collected in this invention have at least 10,000 stars and a large number of fork records, indicating their high quality. Furthermore, this invention extensively collects code edits and changes from pull requests, gathering over 1.8 million code edit histories from more than 93,000 pull requests. Notably, the data collection process of this invention collected a large number of code change samples for pre-training using only 15 open-source projects.

[0051] II. Design of Code Editing Pre-training

[0052] This invention views the development of code editing pre-trained models as a combination of the following three parts: model structure, input representation, and pre-training objective.

[0053] 1) Model Structure

[0054] The model design in this invention is based on the Transformer Encoder-Decoder framework. Building upon this, the invention further pre-trains on the model weights of CodeT5, allowing the model to benefit from the knowledge and representations learned by CodeT5, which is particularly advantageous in code-related tasks.

[0055] 2) Input representation

[0056] To adapt to different types of code evolution and editing tasks, the editing pre-training model proposed in this invention accepts two types of input: type 1) input containing only code; type 2) input containing both editing intent in natural language form and the code to be edited.

[0057] The formal description of input code is as follows, where o i [CLS] represents the i-th token of the old code, and [SEP] represents special tokens in the input.

[0058] x = [CLS], o1, o2, ..., o n [SEP]

[0059] The formal description of the input code and natural language is as follows. Compared to the previous representation, this input adds natural language before the old code, where w i This represents the i-th word in a natural language.

[0060] x = [CLS], w1, w2, ..., w m [SEP],o1,o2,…,o n [SEP]

[0061] 3) Pre-training objectives

[0062] The pre-training objectives involved in this invention aim to learn basic code editing knowledge and how to correctly generate edited code from old code. To generate edited code, this invention proposes three edit-aware pre-training objectives: code editing target generation, code preservation fragment masking, and code editing combined with denoising autoencoders. For example... Figure 2 As shown.

[0063] a) Code editing target generation

[0064] This pre-training task treats the code editing process as a text generation task. It randomly masks a span of code tokens in the input and uses a sequence-to-sequence framework to predict the new code sequentially. Based on the characteristics of the code editing data, this invention sets the average span number to 2.5 and the overall percentage of masked tokens to 20%. Unlike self-supervised tasks in the general text generation domain, the goal of this pre-training task for code editing target generation is not only to perceive the masked words but also to generate complete edited code even with the masked words present.

[0065] In a formal sense, the loss of this mission can be described as follows:

[0066]

[0067] Where t represents the current generation step, and n t This represents the t-th token in the generated edited code. Mask ((ETG)) The method of masking is represented, where N represents the maximum length of the generated sequence, and P represents the maximum length of the generated sequence. (θ) θ represents the probability distribution of the target sequence generated by the model, and θ represents the learnable parameters of the model.

[0068] b) Code preservation fragment mask

[0069] The design of the code fragment mask aims to enable the model to perceive which code fragments need to be modified during code editing and which do not. This pre-training objective considers the similarities in code editing. Code often exhibits patterns and structures, so some code fragments may remain unchanged after evolution. This invention employs a method of masking only tokens that remain unchanged before and after editing when processing edited data, allowing the model to better learn frequently shared common parts during code editing, thereby enhancing its ability to understand and generate code. To obtain the shared tokens before and after code changes, this invention first calculates the differences between the new and old code at the token level. Applying this token masking task to real-world code editing datasets involves not only masking shared tokens but also generating modified code after knowing which tokens do not need to be changed. Since the number of shared tokens before and after code changes is less than the total number of tokens in the original code, 30% of the unchanged tokens are masked in this task, aiming to maintain a similar total number of masked tokens as the previous task.

[0070]

[0071] Among them, o ((keep)) This represents a token that remains unchanged before and after code editing, o (change)A token that shows the changes before and after editing.

[0072] c) Code editing combined with denoising autoencoder

[0073] The third pre-training step of this invention involves combining denoising autoencoders with code editing, specifically integrating the characteristics of denoising autoencoders with real-world code editing scenarios. It aims to train the model to accurately generate new code while considering potential noise in older code. Compared to the first two pre-training tasks, this task particularly demands a stronger understanding of code editing from the model. However, this task aligns with real-world development scenarios because the quality of existing code is often not guaranteed in daily development. It reflects the practicality of code editing, where the quality and effectiveness of existing code vary.

[0074] Specifically, the noise types introduced in this pre-training task include code token insertion, token deletion, and token replacement. During the code editing target generation process, this invention requires the model to consider not only the replaced tokens but also scenarios where existing tokens are deleted and the use of useless masks. Unlike existing methods that only require reconstructing the original input, this pre-training objective also aims to generate realistic code editing objects.

[0075]

[0076] in, This represents the old code after noise was introduced.

[0077] d) Combination of pre-training objectives

[0078] This invention treats each pre-training target and its corresponding task equally, and sums the loss values ​​of each task to obtain the overall loss. The final loss function is described as follows:

[0079]

[0080] III. Evaluation of the Technical Effects of the Invention

[0081] Finally, to evaluate the functionality of this invention, three tasks were selected to assess the code evolution and editing capabilities. These tasks are:

[0082] Code Enhancement: This task involves taking flawed code as input and having the model automatically fix the bugs and output correct code. This task can be used to evaluate the ability of an invention to automatically evolve and edit code without natural language guidance. The evaluation dataset includes two versions: small and medium. The small version contains shorter code, while the medium version generally contains longer code.

[0083] Automated code review: This task involves inputting a piece of code and a natural language description of how that code should be changed. The model automatically modifies the existing code according to the requirements of the natural language description. This task is used to evaluate whether, under natural language guidance, the invention can understand the semantics of natural language and edit the code according to the specified requirements.

[0084] Code Translation: This task involves translating code from one programming language into another. This invention selects translation between Java and C++ programming languages ​​for evaluation. This task is used to assess the robustness of the invention in tasks not strongly correlated with code evolution and editing. The evaluation dataset, CodeTrans, includes both Java-to-C++ and C++-to-Java code translations.

[0085] Evaluation results show that in code enhancement tasks, the present invention achieves accuracy rates of 22.93% and 15.68% respectively, surpassing other code pre-trained models related to editing; in automatic code review tasks, it achieves an accuracy rate of 41.21%, significantly exceeding the best achieved rate of 37.19% by other methods; in code translation tasks, the present invention achieves an accuracy rate of 68.5% in Java to C# code translation and 72.10% in C# to Java code translation, also surpassing code pre-trained models of similar scale.

[0086] In summary, code evolution and editing are indispensable parts of a programmer's daily development process, and modifying existing code to obtain new code is an important application scenario. This invention utilizes a novel pre-trained dataset containing over 1.8 million real-world fine-grained code editing samples and, through a novel fine-grained code editing pre-training objective design, obtains a new pre-trained model for code evolution and editing. Based on this model, this invention can help developers automatically modify original code and quickly obtain the required new code, reducing repetitive programming work such as copying, pasting, and modifying, thereby significantly improving programmer development efficiency.

[0087] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for the evolution and editing of automatic code, characterized in that, The method includes: Construct a training dataset, which consists of several pairs of new and old code; A code evolution and editing model is trained on the training dataset based on code editing target generation loss, code preservation fragment masking loss, and code editing loss combined with denoising autoencoder; wherein, the code evolution and editing model is built on the Transformer Encoder-Decoder framework, and the code editing target generation loss... The code preserves fragment mask loss. The code editing loss combined with denoising autoencoder t represents the current generation step. This represents the t-th token in the generated edited code, and N represents the maximum length of the generated sequence. This represents the probability distribution of the target sequence generated by the model. Represents the learnable parameters of the model. This represents a token that remains unchanged before and after code editing. A token representing the changes before and after editing. This represents the old code after noise was introduced; The test code is evolved or edited based on a code evolution and editing model. The construction of the training dataset includes: Projects are selected based on their popularity on GitHub, and the code of the selected projects is used as the old code. Extract the merged pull request from the selected project, and obtain the new code by extracting code editing patches from the pull request; wherein the difference between the original old code and the original new code does not exceed Yes, and the number of tokens changed from the original old code to the original new code does not exceed a certain percentage of the total number of tokens. , It is a positive integer; After filtering the old and new code based on code length, code content, and code changes, the training dataset is obtained.

2. The method according to claim 1, characterized in that, The filtering of old code pairs based on code content includes: Remove keyword tags associated with GitHub commits; Filter out data whose code content does not contain normal code content, and exclude data that consists only of comments and has no code content.

3. The method according to claim 1, characterized in that, The method of training a code evolution and editing model on the training set based on code editing target generation loss, code preservation fragment masking loss, and code editing loss combined with denoising autoencoder includes: The total loss value is obtained by summing the loss values ​​of code editing target generation loss, code preservation fragment masking loss, and code editing loss combined with denoising autoencoder. During the training of each batch, the model parameters are adjusted by minimizing the total loss value to obtain the code evolution and editing model.

4. The method according to claim 1, characterized in that, The evolution of test code based on the code evolution and editing model includes: Formalize the test code as an input representation. ;in, This is the first special marker in the input representation. This is the second special marker in the input representation. This indicates the number of tokens in the code being tested. Indicates the first... A token for a code; Input representation The code evolution and editing model is fed in to obtain the evolution results of the test code.

5. The method according to claim 1, characterized in that, The method of editing the code under test based on the code evolution and editing model includes: The test code and edit intentions, including natural language forms, are formally described as input representations. ;in, This is the first special marker in the input representation. This is the second special marker in the input representation. This indicates the number of lines of code in the tested code. Indicates the first... A token for a code. Indicates the number of words in a natural language. Representing the first in natural language One word; Input representation The code evolution and editing model is fed in to obtain the encoding results of the test code.

6. An automatic code evolution and editing system, characterized in that, The system includes: The training set construction module is used to construct the training dataset, which consists of several pairs of new and old code. The training module is used to train a code evolution and editing model on the training dataset based on the code editing target generation loss, the code preservation fragment masking loss, and the code editing loss combined with denoising autoencoder; wherein, the code evolution and editing model is built on the Transformer Encoder-Decoder framework, and the code editing target generation loss... The code preserves fragment mask loss. The code editing loss combined with denoising autoencoder t represents the current generation step. This represents the t-th token in the generated edited code, and N represents the maximum length of the generated sequence. This represents the probability distribution of the target sequence generated by the model. Represents the learnable parameters of the model. This represents a token that remains unchanged before and after code editing. A token representing the changes before and after editing. This represents the old code after noise was introduced; The testing module is used to evolve or edit test code based on a code evolution and editing model. The construction of the training dataset includes: Projects are selected based on their popularity on GitHub, and the code of the selected projects is used as the old code. Extract the merged pull request from the selected project, and obtain the new code by extracting code editing patches from the pull request; wherein the difference between the original old code and the original new code does not exceed Yes, and the number of tokens changed from the original old code to the original new code does not exceed a certain percentage of the total number of tokens. , It is a positive integer; After filtering the old and new code based on code length, code content, and code changes, the training dataset is obtained.

Images