A code development system and method with self-debugging capability

Abstract

The application discloses a code development system and method with self-debugging capability, and relates to the technical field of artificial intelligence and software engineering. The system comprises a code defect acquisition module, a self-debugging model module, a verifiable evaluation module and a strategy updating module. The self-debuging model module generates a candidate code repair scheme based on a large language model. The strategy updating module adopts an adaptive entropy-guided reinforcement learning method, calculates the change amount of the model strategy entropy, dynamically adjusts the advantage function used for training, and stimulates the model to intelligently balance between exploring new repair paths and utilizing known effective methods. The corresponding method performs model training and code repair based on the system. The application solves the problems of insufficient exploration capability and premature convergence of repair strategies in the existing code debugging method based on reinforcement learning, and can improve the diversity and robustness of the repair scheme generated by the model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and software engineering technology, specifically to a code development system and method with self-debugging capabilities. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly being used in code generation. AI-assisted programming tools such as GitHub Copilot and Amazon CodeWhisperer have gradually become important tools for improving software development efficiency. These tools can automatically generate code snippets, functions, and even complete modules based on the developer's natural language descriptions or existing code context. However, existing tools are still insufficient when faced with complex logical errors or runtime defects. When bugs exist in the code generated by the model, manual intervention is often still required for debugging, failing to truly achieve full automation of the development process.

[0003] To overcome the aforementioned limitations, those skilled in the art have attempted to endow models with the ability to repair and self-debug code using reinforcement learning. The basic idea is to model the code repair process as a sequential decision problem: the model takes the defective code and its associated compile-time or runtime error information as its state, outputs the repaired code as its action, and then verifies the repair result through a compiler and unit test suite, providing a corresponding reward signal based on the verification result. In recent years, reinforcement learning based on verifiable rewards has made some progress in this direction. This method utilizes an externally determined verification mechanism to perform a binary evaluation on the complete repair trajectory generated by the model: if the repair is successful, a reward of 1 is given; otherwise, 0 is given. Although this reward signal is relatively sparse, it possesses high reliability and determinism.

[0004] However, while this method can guide the model to quickly master validated code repair patterns, new problems have emerged in practice: existing training mechanisms, while rewarding the model for "making the correct repair," have failed to effectively incentivize it to maintain necessary "exploratory behavior." The model lacks an intrinsic, adaptive guidance during training that enables it to intelligently balance "safely adopting known effective methods" with "boldly trying new, potentially better paths." This lack of dynamic balance between exploration and utilization drives the model's generation strategy to prematurely converge to a few validated repair paths, resulting in reduced output diversity. In evaluation, this manifests as a potential increase in the model's "first-attempt repair success rate" (pass@1), but a decrease in its overall ability to "find the correct solution in multiple attempts" (pass@k, k>1), reflecting that the model sacrifices its ability to "explore" better or more robust code repair solutions when "utilizing" known successful patterns. This phenomenon is widespread, and existing research (such as the paper "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?") has been verified through benchmark tests in different fields and model families. Summary of the Invention

[0005] This invention proposes a code development system and method with self-debugging capabilities. Its purpose is to improve the diversity, robustness, and overall success rate of the code repair schemes generated by the model by introducing an adaptive intrinsic guidance mechanism that enables the model to intelligently weigh "using known effective repair methods" and "trying new and better repair paths" during the training process.

[0006] The technical solution of this invention is as follows:

[0007] A code development system with self-debugging capabilities includes a code generation module, which is based on a large language model and is a self-debugging model module; it also includes a code defect acquisition module, a verifiable evaluation module, and a strategy update module.

[0008] The code defect acquisition module is used to receive the source code file to be debugged and obtain error report information related to the source code from the compiler or static code analysis tool;

[0009] The self-debugging model module is used to receive the source code and the error report information output by the code defect acquisition module, and generate one or more candidate code repair solutions;

[0010] The verifiable evaluation module is used to compile and execute each candidate code fix solution, and verify it through a preset set of unit test cases. Based on the verification results, a quantifiable reward signal is generated for each code fix solution.

[0011] The policy update module is used to update the parameters of the large language model in the self-tuning model module according to the reward signal using an adaptive entropy-guided reinforcement learning method.

[0012] As a further improvement to the self-debugging code development system, the adaptive entropy-guided reinforcement learning method adopted by the policy update module includes the following steps:

[0013] Step P1: Calculate the policy entropy of the model under the current code generation strategy, and obtain the change in policy entropy;

[0014] Step P2: Calculate the advantage function guided by adaptive entropy based on the change in policy entropy;

[0015] Step P3: Calculate the loss and update the model parameters based on the advantage function guided by adaptive entropy.

[0016] As a further improvement to the self-debugging code development system, the change in strategy entropy... The calculation method is as follows:

[0017] ;

[0018] In the above formula, The policy entropy of the self-debugging model module under the current code generation strategy. This is the policy entropy from the last update iteration.

[0019] As a further improvement to the self-debugging code development system: for any update iteration, the policy entropy... The calculation formula is:

[0020] ;

[0021] In the above formula, This represents the total number of code fixes generated by the self-debugging model module during this update. This indicates the average length of the code fix solutions, i.e., the average number of time steps contained in each solution; Indicates the first The code fix is ​​in the [number]th [section / ... The state at each time step, namely the code context and error report information at that time; Indicates at time step The model generates one possible action, and at each time step, it selects a final action that will affect the state. This represents the summation of the computational objects corresponding to all possible actions at the current time step; This indicates that the current parameter in the self-tuning model module is... Large language models, in state Next generation action The probability of.

[0022] As a further improvement to the self-debugging code development system: In step P2, let the code repair scheme generated by the self-debugging model module during this update be... , No. Code fixes The reward signal obtained through the verifiable evaluation module is denoted as If the code fix is ​​verified, then ;otherwise ;

[0023] The method for calculating the advantage function guided by adaptive entropy is as follows:

[0024] Step P2-1: Calculate the average reward for all code fixes in this iteration. and the original advantage function for each code fix. ;

[0025] Step P2-2: Based on the original advantage function and the change in the entropy of the strategy Calculate the advantage function of adaptive entropy guidance for each code fix scheme. .

[0026] As a further improvement to the self-debugging code development system, the advantage function of adaptive entropy guidance... The calculation formula is:

[0027] ;

[0028] In the above formula, For the first The advantages of adaptive entropy-guided functions for code fixes; For the first The original advantage function of the code fix solution, It is the absolute value of the original dominance function; and These are preset hyperparameters; It is the hyperbolic tangent function.

[0029] As a further improvement to the self-debugging code development system, step P3 specifically includes:

[0030] Step P3-1: Let the code repair solution generated by the self-debugging model module during this update be... Construct the objective function to be maximized :

[0031] ;

[0032] In the above formula, It is the importance sampling ratio; It is a truncation function used to restrict... In the interval Inside; For hyperparameters; For the first The advantages of adaptive entropy-guided functions for code fixes;

[0033] Step P3-2: Construct the loss function for gradient descent optimization Minimize this loss function using the optimizer. The parameters of the large language model in the self-tuning model module are updated to complete one training iteration.

[0034] As a further improvement to the aforementioned code development system with self-debugging capabilities The calculation formula is:

[0035] ;

[0036] In the above formula, This indicates the average length of the code fix solution. This indicates that the large language model of the self-tuning model module is in the [number]th iteration during this iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This indicates that the large language model of the self-tuning model module in the previous iteration is at the [number]th iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This represents the parameters of the large language model in this iteration. This represents the parameters of the large language model in the previous iteration.

[0037] A code development method with self-debugging capabilities, based on the aforementioned code development system with self-debugging capabilities, includes the following steps:

[0038] Step S1: Training the self-debugging model module: Based on a training dataset containing a large number of defective code samples, the large language model in the self-debugging model module is trained in multiple rounds using the code defect acquisition module, the verifiable evaluation module, and the policy update module.

[0039] Step S2: Repair the code using the trained self-debugging model module: Input the code to be repaired and the corresponding error report information into the trained self-debugging model module, which generates multiple candidate code repair schemes; then verify all candidate code repair schemes through the verifiable evaluation module; and take the verified code repair scheme as the repair result.

[0040] As a further improvement to the self-debugging code development method, each iteration in step S1 performs the following steps:

[0041] Step S1.1: Sample a defective code sample from the training dataset and input it into the code defect acquisition module to obtain error report information;

[0042] Step S1.2: Input the defect code sample and error report information into the self-debugging model module to generate... A variety of code fixes;

[0043] Step S1.3: The verifiable evaluation module compiles and executes each code repair solution separately, and verifies it through a preset set of unit test cases. Based on the verification results, it generates a corresponding reward signal for each code repair solution.

[0044] Step S1.4: The policy update module calculates the loss function and updates the large language model parameters of the self-tuning model module based on the reward signal using an adaptive entropy-guided reinforcement learning method.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention introduces an adaptive entropy-guided reinforcement learning method to construct an intrinsic exploration incentive and exploitation inhibition mechanism within a self-tuning model module based on a large language model. This method adaptively adjusts the reward signal according to the dynamic changes in the model's policy entropy, thereby prompting the model to intelligently balance "adopting known effective repair paths" with "trying new, potentially better solutions" during training. This solves the technical problem in existing technologies where models sacrifice repair path diversity due to an excessive pursuit of validation, resulting in insufficient exploration ability and decreased robustness.

[0047] 2. This invention proposes an adaptive entropy-guided advantage function calculation method based on policy entropy change, which combines the entropy change with the original advantage function after mapping it through a nonlinear function. This method enhances the reward for successful repairs and weakens the penalty for failed repairs when the model explores (entropy increases), and conversely, moderately reduces the reward and correspondingly strengthens the penalty when the model utilizes (entropy decreases). This design provides a quantifiable and differentiable guiding signal for the model to maintain a dynamic balance between exploration and utilization during training, effectively preventing the policy from prematurely converging to a single mode and improving the model's comprehensive ability to find the correct repair solution through multiple attempts.

[0048] 3. This invention constructs a complete closed-loop system comprising modules for code defect acquisition, a self-debugging model, verifiable evaluation, and policy update. The system obtains accurate error reports through a compiler or static analysis tool, generates multiple candidate fixes using a large language model, and generates verifiable reward signals through compilation execution and unit testing. This closed-loop architecture not only tightly integrates the feedback mechanism of reinforcement learning with the specific task of code debugging, but also provides the necessary state, action, and reward data flow for the implementation of the adaptive entropy-guided method, ensuring the engineering feasibility and training stability of the technical solution.

[0049] 4. The system and method proposed in this invention can generate more diverse and higher-quality candidate repair solutions for a given defective code based on the training-derived balancing strategy during the inference stage, and automatically select effective results through a verifiable evaluation module. This not only reduces the cost of manual intervention by developers in the debugging process and improves the automation level of software defect repair, but also lays the foundation for building a more intelligent and autonomous software development auxiliary system. Attached Figure Description

[0050] Figure 1 A flowchart illustrating the training phase of a code development system with self-debugging capabilities.

[0051] Figure 2 A flowchart illustrating the inference phase of a code development system with self-debugging capabilities. Detailed Implementation

[0052] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] A code development system with self-debugging capabilities includes a code defect acquisition module, a self-debugging model module, a verifiable evaluation module, and a strategy update module.

[0054] The code defect acquisition module is used to receive the source code file to be debugged and obtain error report information related to the source code from the compiler or static code analysis tool. The error report information includes error type, error location and error stack.

[0055] The self-debugging model module is based on a large language model and is used to receive the source code and the error report information output by the code defect acquisition module, and generate one or more candidate code repair solutions.

[0056] The verifiable evaluation module is used to compile and execute each candidate code fix solution, and verify it through a preset set of unit test cases. Based on the verification results, a quantifiable reward signal is generated for each code fix solution.

[0057] The policy update module is used to update the parameters of the large language model in the self-tuning model module according to the reward signal using an adaptive entropy-guided reinforcement learning method.

[0058] The steps of the adaptive entropy-guided reinforcement learning method include:

[0059] Step P1: Calculate the policy entropy of the model under the current code generation strategy, and obtain the change in policy entropy.

[0060] Specifically, the strategy entropy of the self-debugging model module under the current code generation strategy is... Policy entropy compared to the last update iteration The difference is used as the change in policy entropy. The calculation formula is as follows:

[0061] .

[0062] Policy entropy measures the overall diversity of all code fixes generated by the model in the current policy update iteration. A higher entropy indicates that the model is more inclined to explore different fix approaches. For any given update iteration, the policy entropy... The calculation formula is:

[0063] ;

[0064] In the above formula, This represents the total number of code fixes generated by the self-debugging model module during this update. This indicates the average length of the code fix plan, i.e., the average number of time steps contained in each plan. Indicates the first The code fix is ​​in the [number]th [section / ... The state at each time step, i.e., the code context and error report information at that time. Indicates at time step The model generates a possible action (corresponding to a code word), and at each time step, it selects a final action that will affect the state. This indicates that the computational objects corresponding to all possible actions at the current time step are summed. This indicates that the current parameter in the self-tuning model module is... Large language models, in state Next generation action The probability of.

[0065] Step P2: Calculate the advantage function guided by adaptive entropy based on the change in policy entropy.

[0066] Let the code fix generated by the self-debugging model module during this update be: , No. Code fixes The reward signal obtained through the verifiable evaluation module is denoted as If the code fix is ​​verified, then ;otherwise .

[0067] Step P2-1: Calculate the average reward for all code fixes in this iteration. and the original advantage function for each code fix. .

[0068] Step P2-2: Based on the original advantage function and the change in the entropy of the strategy Calculate the advantage function of adaptive entropy guidance for each code fix scheme. :

[0069] ;

[0070] In the above formula, For the first The advantages of adaptive entropy-guided functions for code fixes; For the first The original advantage function of the code fix solution, It is the absolute value of the original advantage function, which makes the magnitude of the entropy-guided adjustment proportional to the importance of the original code repair solution; and These are preset hyperparameters. Used to control the overall strength of entropy guidance, Used to control sensitivity to changes in entropy. It is the hyperbolic tangent function, which will Nonlinear mapping to Within the range, the strength of the guidance is correlated with the degree of entropy change, thus avoiding drastic abrupt changes.

[0071] The working mechanism of the advantage function guided by adaptive entropy is as follows:

[0072] when and This means that a successful code fix arises during the model's "exploration" process. This will increase in order to encourage models to generate a variety of effective strategies.

[0073] when and This means that a successful code fix arises from the model "utilizing" existing experience. This will be reduced to moderately punish over-exploitation and prevent the model from prematurely converging to a single repair mode.

[0074] when and This means that a failed code fix arose during the model's "exploration" process. The negative value will decrease, meaning the penalty will be weakened, to encourage the model to boldly try new repair ideas.

[0075] when and This means that a failed code fix arises from the model's "utilization" of existing experience. The negative value of will increase, that is, the penalty is enhanced, in order to suppress the model from making mistakes in deterministic states.

[0076] Step P3: Calculate the loss and update the model parameters based on the advantage function guided by adaptive entropy.

[0077] Step P3-1: Let the code repair solution generated by the self-debugging model module during this update be... Construct the objective function to be maximized :

[0078] ;

[0079] In the above formula, It is the importance sampling ratio, calculated as the total code fix solution generated under the new code generation strategy and the old code generation strategy. The ratio of the probabilities; It is a truncation function used to restrict... In the interval Inside; This is a hyperparameter used to limit the magnitude of policy updates, preventing the new policy from deviating too far from the old policy and causing training instability. For the first The advantage function of adaptive entropy guidance for a code fix scheme.

[0080] Furthermore, The calculation formula is:

[0081] ;

[0082] In the above formula, This indicates the average length of the code fix solution. This indicates that the large language model of the self-tuning model module is in the [number]th iteration during this iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This indicates that the large language model of the self-tuning model module in the previous iteration is at the [number]th iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This represents the parameters of the large language model in this iteration. This represents the parameters of the large language model in the previous iteration.

[0083] Step P3-2: Construct the loss function for gradient descent optimization Minimize this loss function using the optimizer. The parameters of the large language model in the self-tuning model module are updated to complete one training iteration.

[0084] A code development method with self-debugging capabilities, comprising the following steps:

[0085] Step S1: Train the self-tuning model module.

[0086] like Figure 1 As shown, the large language model in the self-debugging model module is trained iteratively through multiple rounds using a training dataset containing a large number of defective code samples. The following sub-steps are performed in each iteration:

[0087] Step S1.1: Sample a defective code sample from the training dataset and input it into the code defect acquisition module to obtain error report information.

[0088] The code defect detection module can be an integrated development environment (IDE) plugin that automatically calls the compiler or linter to capture returned error reports, such as "Segmentation fault (core dumped)" or "IndexError: list index out of range".

[0089] Step S1.2: Input the defect code sample and error report information into the self-debugging model module to generate... A variety of code fixes.

[0090] Step S1.3: The verifiable evaluation module compiles and executes each code repair solution separately, verifies it using a preset set of unit test cases, and generates a corresponding reward signal for each code repair solution based on the verification results.

[0091] Step S1.4: The policy update module calculates the loss function and updates the large language model parameters of the self-tuning model module based on the reward signal using the adaptive entropy-guided reinforcement learning method (steps P1 to P3 above).

[0092] After millions of iterations of training, the entire system can be deployed.

[0093] Step S2: Use the trained self-debugging model module to fix the code.

[0094] The code to be repaired and the corresponding error report information are input into the trained self-debugging model module, which generates multiple candidate code repair solutions. Then, the verifiable evaluation module verifies all candidate code repair solutions. Finally, the verified code repair solutions are presented to the developers or used directly to replace the defective parts in the source code.

[0095] It should be noted that, as will be apparent to those skilled in the art, the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics thereof. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A code development system with self-debugging capabilities, comprising a code generation module, wherein the code generation module is based on a large language model, characterized in that: The code generation module is a self-debugging model module; it also includes a code defect acquisition module, a verifiable evaluation module, and a strategy update module. The code defect acquisition module is used to receive the source code file to be debugged and obtain error report information related to the source code from the compiler or static code analysis tool; The self-debugging model module is used to receive the source code and the error report information output by the code defect acquisition module, and generate one or more candidate code repair solutions; The verifiable evaluation module is used to compile and execute each candidate code fix solution, and verify it through a preset set of unit test cases. Based on the verification results, a quantifiable reward signal is generated for each code fix solution. The policy update module is used to update the parameters of the large language model in the self-tuning model module according to the reward signal using an adaptive entropy-guided reinforcement learning method; The adaptive entropy-guided reinforcement learning method used in the policy update module includes the following steps: Step P1: Calculate the policy entropy of the model under the current code generation strategy, and obtain the change in policy entropy; Change in policy entropy The calculation method is as follows: ； In the above formula, The policy entropy of the self-debugging model module under the current code generation strategy. The policy entropy from the last update iteration; For any given update iteration, the policy entropy The calculation formula is: ； In the above formula, This represents the total number of code fixes generated by the self-debugging model module during this update. This indicates the average length of the code fix solutions, i.e., the average number of time steps contained in each solution; Indicates the first The code fix is ​​in the [number]th [section / ... The state at each time step, namely the code context and error report information at that time; Indicates at time step The model generates one possible action, and at each time step, it selects a final action that will affect the state. This represents the summation of the computational objects corresponding to all possible actions at the current time step; This indicates that the current parameter in the self-tuning model module is... Large language models, in state Next generation action The probability of; Step P2: Calculate the advantage function guided by adaptive entropy based on the change in policy entropy; In step P2, let the code repair scheme generated by the self-debugging model module during this update be... , No. Code fixes The reward signal obtained through the verifiable evaluation module is denoted as If the code fix is ​​verified, then ;otherwise ; The method for calculating the advantage function guided by adaptive entropy is as follows: Step P2-1: Calculate the average reward for all code fixes in this iteration. and the original advantage function for each code fix. ; Step P2-2: Based on the original advantage function and the change in the entropy of the strategy Calculate the advantage function of adaptive entropy guidance for each code fix scheme. ; Advantage function guided by adaptive entropy The calculation formula is: ； In the above formula, For the first The advantages of adaptive entropy-guided functions for code fixes; For the first The original advantage function of the code fix solution, It is the absolute value of the original dominance function; and These are preset hyperparameters; It is the hyperbolic tangent function Step P3: Calculate the loss and update the model parameters based on the advantage function guided by adaptive entropy; Step P3 specifically includes: Step P3-1: Let the code repair solution generated by the self-debugging model module during this update be... Construct the objective function to be maximized : ； In the above formula, It is the importance sampling ratio; It is a truncation function used to restrict... In the interval Inside; For hyperparameters; For the first The advantages of adaptive entropy-guided functions for code fixes; The calculation formula is: ； In the above formula, This indicates the average length of the code fix solution. This indicates that the large language model of the self-tuning model module is in the [number]th iteration during this iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This indicates that the large language model of the self-tuning model module in the previous iteration is at the [number]th iteration. The state at each time step The next action selected at this time step will be generated. The probability of; This represents the parameters of the large language model in this iteration. This represents the parameters of the large language model in the previous iteration; Step P3-2: Construct the loss function for gradient descent optimization Minimize this loss function using the optimizer. The parameters of the large language model in the self-tuning model module are updated to complete one training iteration.

2. A code development method with self-debugging capabilities, characterized in that, This method is based on the code development system with self-debugging capabilities as described in claim 1, and includes the following steps: Step S1: Training the self-debugging model module: Based on a training dataset containing a large number of defective code samples, the large language model in the self-debugging model module is trained in multiple rounds using the code defect acquisition module, the verifiable evaluation module, and the policy update module. Step S2: Repair the code using the trained self-debugging model module: Input the code to be repaired and the corresponding error report information into the trained self-debugging model module, which generates multiple candidate code repair schemes; then verify all candidate code repair schemes through the verifiable evaluation module; and take the verified code repair scheme as the repair result.

3. The code development method with self-debugging capability as described in claim 2, characterized in that, In each iteration of step S1, the following steps are performed: Step S1.1: Sample a defective code sample from the training dataset and input it into the code defect acquisition module to obtain error report information; Step S1.2: Input the defect code sample and error report information into the self-debugging model module to generate... A variety of code fixes; Step S1.3: The verifiable evaluation module compiles and executes each code repair solution separately, and verifies it through a preset set of unit test cases. Based on the verification results, it generates a corresponding reward signal for each code repair solution. Step S1.4: The policy update module calculates the loss function and updates the large language model parameters of the self-tuning model module based on the reward signal using an adaptive entropy-guided reinforcement learning method.

Images