Unit test case generation large model fine tuning method based on feedback reinforcement learning

Abstract

The invention relates to the technical field of software testing, and discloses a unit test case generation large model fine tuning method based on feedback reinforcement learning, which comprises the following steps: constructing a unit test fine tuning data set based on demand driving; an Actor model is constructed, and optimization is carried out through a demand-driven unit test fine tuning data set; a Critic model is constructed, and optimization is carried out through a demand-driven unit test fine tuning data set; and generating a unit test case based on the optimized Actor model, mapping the unit test case into a single instant reward based on the optimized Critic model, and obtaining a large model generated based on the unit test case through reinforcement learning. The unit test case generation method capable of automatically generating high-quality unit test cases with aligned requirements and consistent styles is constructed, the test efficiency and quality are remarkably improved, and the bottlenecks of traditional test generation in manual dependence, coverage depth and requirement consistency are broken through.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more specifically, to a large model fine-tuning method for generating unit test cases of human feedback reinforcement learning based on requirement-driven. Background Art

[0002] With the continuous increase in the scale and iteration frequency of software systems, unit testing has become a key link to ensure code quality, inhibit defect spread, and shorten the regression cycle. However, in practice, the writing of test scripts still mainly relies on developers or test engineers to complete manually. They must first thoroughly understand the business requirements, and then write assertions and input data according to function interfaces and potential boundary conditions. Facing complex logic or frequently evolving requirements, this process is time-consuming and laborious and is extremely prone to lag behind code updates, resulting in insufficient coverage and regression omissions. Traditional automated generation techniques, such as symbolic execution, constraint solving, search testing, and unit test generation based on static templates, although can produce test cases in batches to a certain extent, often focus on syntactic legality, statement coverage, and branch coverage, and pay insufficient attention to the semantic correctness of assertions, business correctness, and code specifications; the generated results still require a large amount of manual post-revision, and the overall benefit is limited.

[0003] In recent years, large language models (LLMs) have demonstrated powerful code understanding and generation capabilities. Some studies have tried to use supervised fine-tuning (SFT) methods to let large models complete or generate unit test cases. However, the quality of test scripts in public datasets is uneven, and the "maximum likelihood" training objective cannot directly optimize metrics that can only be observed after execution, such as assertion authenticity and path coverage. The model is prone to copying data noise and generating code fragments that do not conform to the project style. In addition, in industrial scenarios, test cases often must echo high-level requirement documents, and it is difficult to capture these requirement semantics only based on function signatures and interface annotations, resulting in the disconnection between the generated content and the real business requirements.

[0004] Human feedback reinforcement learning (RLHF) has been proven to significantly improve the output quality in dialogue optimization and code completion tasks: first, train a reward model through "Chosen - Rejected" examples, and then use proximal policy optimization algorithms (such as PPO) to fine-tune the large model to make its output more in line with human preferences. Applying RLHF directly to unit test generation faces new challenges: first, the test execution can only obtain a one-time overall reward after the script runs, and the reward signal is extremely sparse; second, a high-quality test script needs to meet multiple requirements such as assertion correctness, path or branch coverage, and code style specifications at the same time, and there are often trade-offs between these goals; third, requirement-driven testing emphasizes that test cases must trace and meet business requirements, but how to map requirement semantics into differentiable reward and punishment signals and continuously inject them during the RLHF process is an unsolved problem.

[0005] Current research thus presents three breaks: automated test methods are difficult to accurately align with requirements, supervised fine-tuning of LLMs cannot directly optimize execution metrics, and existing RLHF applications lack fine-grained reward design and requirement constraints for test tasks. To bridge the three, a new technical framework is needed: it can not only make full use of the innate advantages of LLMs in code generation, but also transform complex test quality metrics into learnable signals by decomposing the reward function, and absorb business requirement changes and human expert feedback in real time during the training loop, so that the generated unit tests are synchronized with code evolution and consistent with business intentions. A large model fine-tuning method for generating unit test cases based on feedback reinforcement learning is proposed to break through the bottlenecks of existing automated tests in terms of labor cost, coverage depth, and requirement consistency. Summary of the Invention

[0006] In view of this, the present invention provides a large model fine-tuning method for generating unit test cases based on feedback reinforcement learning, which overcomes the problem that traditional LLM supervised fine-tuning only relies on code context and cannot accurately map high-level requirements, resulting in the disconnection between the generated test scripts and real business intentions, resolves the limitation that a single maximum likelihood training objective cannot directly optimize execution metrics such as assertion correctness, branch coverage, and code specifications, eliminates the problem that existing RLHF is unstable in training due to sparse rewards, difficult credit assignment, and policy divergence in test tasks, and ensures stable convergence through advantage estimation and KL divergence constraints; enables test cases to evolve in real time with software iteration, significantly reducing the labor maintenance cost and improving test coverage and assertion effectiveness.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A large model fine-tuning method for generating unit test cases based on demand-driven human feedback reinforcement learning, comprising:

[0009] Construct a demand-driven unit test fine-tuning dataset;

[0010] Construct an Actor model and optimize it through a demand-driven unit test fine-tuning dataset;

[0011] Construct a Critic model and optimize it through a demand-driven unit test fine-tuning dataset;

[0012] Generate unit test cases based on the optimized Actor model, map the unit test cases to a single immediate reward based on the optimized Critic model, and obtain a large model based on unit test case generation through reinforcement learning.

[0013] Preferably, construct a unit test fine-tuning dataset driven by requirements, including:

[0014] Obtain the dataset and clean the dataset;

[0015] Add a Rejected part to each training data in the dataset, perform assertion truth value error mutation, answer duplication mutation, and format error mutation on each training data to construct the Rejected class, and obtain a unit test fine-tuning dataset driven by requirements. Each data in the unit test fine-tuning dataset driven by requirements is a quadruple (Prompt, Chosen, Rejected, Response). Prompt is the prompt word, which includes the code to be tested and functional requirement information. Chosen is the result that the large model is expected to return under the corresponding Prompt. Rejected is the class of results that the large model is not expected to return under the corresponding Chosen. Response is the true result generated by the large model.

[0016] Preferably, the Rejected class includes the large model-generated assertion truth value error class, the large model-generated test case duplication class, and the large model-generated test case format error class.

[0017] Preferably, construct an Actor model and optimize it through a unit test fine-tuning dataset driven by requirements, including:

[0018] Use the OPT-1.3B large model as the backbone network and insert the LoRA layer in the attention layer;

[0019] Perform supervised fine-tuning on the processed OPT-1.3B large model based on the Prompt-Chosen pairs in the unit test fine-tuning dataset driven by requirements, with the NLL loss function as the objective, to obtain the optimized Actor model.

[0020] Preferably, the NLL loss function is:

[0021]

[0022] where represents the set of trainable parameters of the model, including the backbone Transformer weights and lightweight parameters, represents the conditional context, that is, tell the large model which content to generate unit test cases for through the Prompt, represents the true token at position i of the target sequence, that is, the correct answer, represents the prefix token sequence , and only look at the previously generated token sequence when predicting the next token. The conditional probability given by the representative model, that is, the probability of generating the current token when the model knows the Prompt content and the token sequence content before the token to be generated, represents the target sequence length.

[0023] Preferably, construct a Critic model and optimize it through a requirement-driven unit test fine-tuning dataset, including:

[0024] Use the OPT-1.3B weights as the encoder;

[0025] Freeze the backbone weights and insert a LoRA layer in the attention mechanism;

[0026] Connect three different token-wise scalar heads to the output end of the encoder to score the hidden state of each token:

[0027]

[0028] Among them, is the hidden vector of the t-th token, is the original score of this token, represents the projection matrix, and b is the bias term;

[0029] Model the scoring difference between Chosen and Rejected in the requirement-driven unit test fine-tuning dataset based on pairwise ranking loss, and at the same time minimize the three-way token-wise scalar head loss to obtain the optimized Critic model, including the assertion truth value error model, the answer repetition model, and the format error model.

[0030] Preferably, the pairwise ranking loss is:

[0031]

[0032] Among them, represents the overall loss of the Critic model, N is the target sequence length, that is, the total number of tokens, t is the token position index, from Chosen is the original score of the t-th token in Chosen by the Critic model, from Rejected is the original score of the t-th token in the Rejected script by the Critic model, represents the Sigmoid function, which maps the difference to the interval (0,1). ​

[0033] Preferably, unit test cases are generated based on the optimized Actor model, and the unit test cases are mapped to a single immediate reward based on the optimized Critic model. The large model generated based on the unit test cases is obtained through reinforcement learning, including:

[0034] Taking the Prompt as the initial state and autoregressively generating unit test cases through the optimized Actor model;

[0035] Inputting the unit test cases into the optimized Critic model, scoring each token in the complete test case, and fusing them into a single immediate reward:

[0036]

[0037] Among them, , , is the three-way score, is the single immediate reward, , , represent the weight coefficients;

[0038] Calculating the discounted cumulative return based on the single immediate reward:

[0039]

[0040] Among them, is the discounted cumulative return, t refers to the current time step or token position, 1 ≤ t ≤ T, T refers to the termination time step of a complete generation process, refers to the single immediate reward obtained at time step k, represents the discount factor;

[0041] Calculating the advantage function based on the discounted cumulative return:

[0042]

[0043] Among them, is the scalar estimate of the state by the optimized Critic model, is the advantage function;

[0044] Guiding parameter updates based on the advantage function, performing policy updates on the optimized Actor model through PPO, performing value estimation updates on the optimized Critic model using A2C, and adding KL penalty constraints to the policy. Iterate the above process until the target metrics converge to obtain the large model generated based on unit test cases.

[0045] Preferably, the large model joint loss function generated based on unit test cases is:

[0046]

[0047] Among them, represents the joint loss function, represents the Policy loss, represents the value loss, represents the KL penalty constraint, represents the weight of represents the weight of;

[0048]

[0049]

[0050] Among them, represents the probability of generating action at state in this round of iteration, represents the probability of generating action at state in the previous round of iteration, represents the importance sampling ratio, represents the advantage function, represents width, restricting policy drift, represents truncating the importance sampling ratio in the interval ; represents averaging a batch of sampled according to the current policy at time step t;

[0051]

[0052] Among them, represents the current predicted value of the optimized Critic model, represents the predicted value of the previous iteration, represents the discounted cumulative return, represents the truncation function;

[0053]

[0054] Among them, represents the Reference Model, represents the weight coefficient, represents the asymmetric index for measuring the distance between two distributions.

[0055] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a fine-tuning method for a large model for generating unit test cases based on feedback reinforcement learning, which has the following effects:

[0056] 1) The requirement consistency is significantly improved

[0057] The requirement parser is used to automatically convert the function description or defect description of the module under test into "requirement embedding", which runs through the entire process of Actor-Critic reward calculation, making the generated test script highly consistent with the real business requirements in terms of assertion intention, input scenario and boundary conditions, and enabling direct end-to-end verification of the business logic, reducing regression defects caused by missed requirements.

[0058] 2) The test quality leaps under the drive of multi-metric differentiable rewards

[0059] Dimensions such as assertion correctness, branch / path coverage, and code normativity are decomposed into differentiable sub-rewards and fused into an adaptive weighted total reward; obtaining dense gradient feedback for each token helps the generated unit test cases to approach the industry-leading level in terms of coverage depth, assertion effectiveness and maintainability.

[0060] 3) The training converges more stably and has high sample efficiency

[0061] Proximal Policy Optimization (PPO-Clip) is combined with value network clipping and KL reference model constraint to solve the problem of policy divergence caused by sparse rewards in test scenarios; token-level advantage estimation alleviates the long sequence credit assignment problem, enabling the model to converge stably within a few rounds.

[0062] 4) Parameter-efficient and low-cost continuous iteration

[0063] Through the LoRA lightweight module, only <2% of the parameters need to be updated to complete fine-tuning; when the requirements or code baseline change, only new requirement-test pairs need to be incrementally collected and a small number of RLHF steps need to be executed to quickly realign.

[0064] 5) The generation style is safe and controllable

[0065] Introduce the KL penalty term to align the supervised fine-tuning reference model, effectively suppressing language drift and hallucination output; at the same time, with the help of code normativity sub-rewards, automatically maintain the established code style of the project (such as PEP8, pytest, etc.).

[0066] 6) Comprehensive economic and quality benefits

[0067] Deploying the solution of this invention in a continuous delivery pipeline can reduce the manual cost of writing unit tests by 60–70%, increase the early exposure rate of defects, and significantly shorten the requirement-test closed-loop time by automatically aligning with the latest requirements.

[0068] Through the above technical path, the present invention not only achieves the targeted enhancement of the unit test generation capability of large language models, but also opens up the three key links of demand-driven, quality indicator optimization and sustainable evolution, providing a new paradigm for software testing automation that is efficient, reliable and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0070] Figure 1 This is a flow chart of the method for generating large models from unit test cases based on feedback reinforcement learning provided by the present invention.

[0071] Figure 2 This is a block diagram of the method for generating large models from unit test cases based on feedback reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] The embodiment of the present invention discloses a method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning, such as Figure 1 Shown, including:

[0074] Build a demand-driven unit test fine-tuning dataset;

[0075] Build the Actor model and fine-tune the dataset for optimization through demand-driven unit testing;

[0076] Build the critic model and optimize it by fine-tuning the dataset through demand-driven unit testing;

[0077] Generate unit test cases based on the optimized Actor model, map the unit test cases to a single immediate reward based on the optimized Critic model, and obtain a large model based on the generation of unit test cases through reinforcement learning.

[0078] The following is a specific description of each step:

[0079] (1) Construct a unit test fine-tuning dataset driven by requirements. This dataset includes 157,767 test data for training. Each data consists of a quadruple (Prompt, Chosen, Rejected, Response). Prompt is the prompt word, which includes the code under test and function requirement information. Chosen is the result that the large model is expected to return under the corresponding Prompt. Rejected is the class of results that the large model is not expected to return under the corresponding Chosen. Rejected is divided into three categories, including the category of incorrect assertion true values generated by the large model, the category of duplicate test cases generated by the large model, and the category of incorrect test case formats generated by the large model. Response is the actual result generated by the large model.

[0080] The specific construction process is as follows:

[0081] Obtain the dataset:

[0082] It can be the "stack-dedup-python-testgen-starcoder-filter-v2" dataset on Hugging Face;

[0083] Clean the dataset, remove the training data with a length greater than 6k in the dataset, remove the training data with a coverage rate lower than 90%, and remove the training data with test case failures;

[0084] Then add the Rejected part to each training data in this dataset. The dataset itself contains the Prompt, Chosen, and Response parts. Concentrate on performing "assertion true value error" mutation, "answer duplication" mutation, and "format error" mutation on each training data to construct the Rejected class, and obtain a unit test fine-tuning dataset driven by requirements.

[0085] (2) Construct an Actor (actor) model and optimize it through a unit test fine-tuning dataset driven by requirements. Its role is to read in the Prompt (the function under test and requirement description) and generate a complete preliminary unit test case.

[0086] Specifically, first, select the OPT-1.3B large model released by Facebook as the backbone. Then freeze most of the original weights and insert LoRA layers in the attention mechanism to significantly reduce the video memory requirements and the number of training steps during the training process. Through the LoRA lightweight module, only <2% of the parameters need to be updated to complete the fine-tuning. When performing supervised fine-tuning, the Prompt and Chosen parts in the first 40% of the training data in the demand-driven unit test fine-tuning dataset can be used. Selecting the first 40% is a low-cost and low-risk step-by-step fine-tuning strategy, which ensures the rapid acquisition of basic capabilities and reserves data for subsequent refined optimization and performance evaluation. Of course, other numerical ratios of training data can also be selected according to actual needs. Through the NLL loss function:

[0087]

[0088] Among them, represents the set of trainable parameters of the model, including the backbone Transformer weights and lightweight parameters such as LoRA weights, etc. represents the conditional context, that is, the Prompt, whose role is to tell the model what content to generate unit test cases for. represents the true token of the target sequence at position i, that is, the correct answer. represents the prefix token sequence , and only look at the previously generated token sequence when predicting the next token. represents the model's pair The conditional probability given, that is, given the content of the Prompt and the token sequence content before the token to be generated, the probability of generating the current token, and it is hoped that it is as close to 1 as possible. represents the length of the target sequence. For each target token, look at the probability given by the model. The lower the probability, the greater the loss, which can force the model to increase the probability of the correct token.

[0089] (3) Build a Critic (scorer) model and optimize it through the demand-driven unit test fine-tuning dataset. Its positioning is to map the unit test cases produced by the Actor into scalar rewards. Train multiple sub-evaluators with pairwise ranking loss to let the model learn to score on the semantics of Chosen and Rejected.

[0090] In specific implementation, the Chosen and Rejected parts in the first 20% - 60% of the training data in the dataset are fine-tuned using requirement-driven unit tests to obtain a more stable and generalization-capable ranking discrimination. Of course, other numerical ratios of training data can also be selected according to actual requirements, and the "Chosen - Rejected" paired training model is used to learn what a "good" generation result is and what a "bad" generation result is, learn generation preferences, and directly reuse the same OPT-1.3B large model weights as in the Actor as the encoder, freeze the backbone weights, insert the LoRA layer in the attention mechanism, and only connect three different token-wise scalar heads at the encoder output. Token-wise means processing each token one by one: the model gives an output for each token (word, subword, or character) in the sequence separately. Scalar head refers to a linear output head: it maps the hidden vector (with dimension d) of the t-th token into a :

[0091]

[0092] where, is the hidden vector of the t-th token, is the original score of this token, represents the projection matrix (also a learnable parameter), which is responsible for compressing the hidden vector to 1 dimension, and b is the bias term. Obtaining dense gradient feedback for each token helps the generated unit test cases to be close to the industry-leading level in terms of coverage depth, assertion validity, and maintainability.

[0093] Based on the pairwise ranking loss, the scoring difference between Chosen and Rejected in the requirement-driven unit test fine-tuned dataset is modeled, and at the same time, the three-way token-wise scalar head loss is minimized to obtain an optimized Critic model, including the assertion truth error model, the answer duplication model, and the format error model.

[0094] Pairwise ranking loss function:

[0095]

[0096] where, represents the overall loss of the Critic model. The smaller the loss, the more the model tends to give a higher score to Chosen, which is also the goal of the present invention. N is the target sequence length, that is, the total number of tokens. t is the token position index. from Chosen It refers to the original score of the t-th token in Chosen by Critic. The higher the score, the better the quality. from Rejected It refers to the original score of the t-th token in the Rejected script by Critic. denotes the Sigmoid function, which maps the difference to the interval (0,1). For each token, if the difference is positive, Sigmoid is close to 1 and log is close to 0. In this case, the loss contribution is small and this situation is encouraged. If the difference is negative, that is then Sigmoid is close to 0, log is negative and has a large absolute value, and the loss contribution is large. The model will penalize this situation from occurring. Finally, averaging ensures that the gradients of long and short scripts are in the same order of magnitude.

[0097] In the three-way token-wise scalar head of the present invention, Critic-A: evaluates whether the assertion passes and whether the execution result meets the expectation (semantic correctness). Its training signals are Pass / Fail (Pass: the test case passes on the code under test, Fail: the test case does not pass on the code under test) labels and symbolic execution results. The key point of the judgment score is whether the assertion is consistent with the semantics of the function under test, and a low score is given to the token at the failed assertion. Critic-B evaluates the coverage rate (path / branch coverage). That is, its training signals are branch and statement coverage rates. The key point of the judgment score is whether the test covers more logic. Using the coverage rate difference as the soft label gradient, positive rewards are given to tokens with high coverage rates. Critic-C evaluates code normativity. For example, its training signal is the flake8 score. The key point of the judgment score is the PEP8 specification, whether there is duplicate code, and maintainability. Negative scores are marked for the tokens where specification errors occur. During training, multi-task learning is adopted, using a backbone and three heads, and training in the way of minimizing the three-way loss at the same time. Finally, adaptive weights are used for fusion and normalization.

[0098] (4)Construct the RLHF (Reinforcement Learning from Human Feedback) module. The role of the RLHF module is to further fine-tune the preliminarily fine-tuned Actor model through reinforcement learning to obtain a large model for generating unit test cases based on demand-driven human feedback reinforcement learning.

[0099] Specifically, first, sampling is carried out. Using the Prompt (function under test + requirement description) in the training set as the initial state, the unit test case token sequence is autoregressively generated through the Actor model, that is, (Policy) ;

[0100] Next is the scoring stage. The generated unit test cases are respectively input into the optimized Critic models, namely Critic-A, Critic-B, and Critic-C models, to obtain three-way scalar scores. , , , which are normalized and then fused into a single immediate reward through learnable weights , , :

[0101]

[0102] Calculate the discounted cumulative return :

[0103]

[0104] Among them, the discounted cumulative return refers to the total reward obtained from time step t to the end of the episode (a complete generation process), which is the result of adding up after being attenuated by the discount factor. t refers to the current time step or token position, where 1 ≤ t ≤ T, and T refers to the termination time step of this episode. refers to the single immediate reward obtained at time step k. represents the discount factor. When is smaller, more emphasis is placed on recent rewards; there is no discount. represents the accumulation of all rewards from time step t to the end point T, and the component coefficient of the discount factor for this step of the reward , and the attenuation is stronger the farther away from the current (the larger k - t is).

[0105] Calculate the advantage function:

[0106]

[0107] Among them, is the scalar estimate of the state by the optimized Critic model. If , it means that this action is better than the expected action, and the probability should be increased in subsequent updates. Otherwise, the probability should be decreased.

[0108] Enter the stage of updating parameters. Use PPO to update the policy of the optimized Actor model, and use A2C to update the value estimate of the optimized Critic model. A2C (Advantage Actor-Critic) is a synchronous policy gradient algorithm: The Actor model gives the action The probability distribution, the Critic network simultaneously estimates the value V(s) of this state, and the two are updated collaboratively; during training, the advantage function is used to replace the original return, which not only indicates whether the action is good or bad relative to the average level but also effectively reduces the gradient variance. At the same time, a KL penalty constraint is added to ensure that the policy does not deviate from the Reference Model. The Reference Model is a benchmark policy frozen before entering the reinforcement learning stage. Iterate the above process until the target metrics (assertion pass rate, coverage rate, format correctness rate) converge.

[0109] Among them, the Actor (Policy) loss function is as follows:

[0110]

[0111]

[0112] Among them, represents the probability of generating action in state in this round of iteration (under the new policy). represents the probability of generating action in state in the previous round of iteration (under the old policy). represents the importance sampling ratio. represents the advantage function. represents width, restricting policy drift, is a truncation function. means truncating the importance sampling ratio in the interval . PPO-Clip allows the policy to maximize these advantages without deviating too far from the old distribution, thus gradually optimizing in the direction of high rewards.

[0113] Value loss function:

[0114]

[0115] Among them, represents the current predicted value of the optimized Critic model, represents the predicted value of the previous iteration, represents the discounted cumulative return, where is the value network clipping.

[0116] KL penalty term:

[0117]

[0118] Among them represents the Reference Model, represents the weight coefficient, represents an asymmetric index for measuring the distance between two distributions. Here, it measures the current policy and the reference policy The average log probability difference between them. The larger the value, the greater the difference between the two. Its role is to prevent language hallucinations and maintain consistency with the writing style of human examples.

[0119] The combined loss function is as follows:

[0120]

[0121] In the backpropagation stage, first regard the advantage as a baton to adjust the policy, and Clip ensures the safety of the update. approximate , and provide a more accurate baseline for the next round. Prevent "going off-topic" and ensure that the output distribution is similar to that of human examples.

[0122] In the RLHF model, the Actor model is updated using PPO Clip, the Critic model is updated using A2C, the Reward Model (benchmark model) uses the same model as the Critic model but does not update the parameters. The ReferenceModel uses the same model as the Actor Model but does not update the parameters.

[0123] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same and similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0124] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-tuning method for a large model of unit test case generation based on feedback reinforcement learning, characterized in that Including: Construct a unit test fine-tuning dataset driven by requirements; Construct an Actor model and optimize it through the unit test fine-tuning dataset driven by requirements; Construct a Critic model and optimize it through the unit test fine-tuning dataset driven by requirements; Generate unit test cases based on the optimized Actor model, map the unit test cases to a single immediate reward based on the optimized Critic model, and obtain a large model based on unit test case generation through reinforcement learning.

2. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 1, wherein Construct a unit test fine-tuning dataset driven by requirements, including: Obtain the dataset and clean it; Add a Rejected part to each training data in the dataset, perform assertion truth value error mutation, answer duplication mutation, and format error mutation on each training data to construct the Rejected class, and obtain the unit test fine-tuning dataset driven by requirements. Each data in the unit test fine-tuning dataset driven by requirements is a quadruple (Prompt, Chosen, Rejected, Response). Prompt is the prompt word, which includes the code under test and functional requirement information. Chosen is the result that the large model is expected to return under the corresponding Prompt. Rejected is the class of results that the large model is not expected to return under the corresponding Chosen. Response is the actual result generated by the large model.

3. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 2, characterized in that, The Rejected class includes the large model-generated assertion truth value error class, the large model-generated test case duplication class, and the large model-generated test case format error class.

4. The fine-tuning method for a large model for generating unit test cases based on feedback reinforcement learning according to claim 2, wherein Construct an Actor model and optimize it through the unit test fine-tuning dataset driven by requirements, including: Use the OPT-1.3B large model as the backbone network and insert a LoRA layer in the attention layer; Perform supervised fine-tuning on the processed OPT-1.3B large model based on the Prompt-Chosen pairs in the unit test fine-tuning dataset driven by requirements, with the NLL loss function as the objective to obtain the optimized Actor model.

5. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 4, wherein NLL loss function is as follows: ； Among them, represents the set of trainable parameters of the model, including the backbone Transformer weights and lightweight parameters, represents the conditional context, that is, telling the large model which content to generate unit test cases for through the Prompt, represents the true token of the target sequence at position i, that is, the correct answer, represents the prefix token sequence , and only looks at the previously generated token sequence when predicting the next token, represents the model's given conditional probability, that is, the probability of generating the current token when the model knows the content of the Prompt and the content of the token sequence before the token to be generated, represents the length of the target sequence.

6. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 2, characterized in that, Construct a Critic model and optimize it through the unit test fine-tuning dataset driven by requirements, including: Use the OPT-1.3B weights as the encoder; Freeze the backbone weights and insert a LoRA layer in the attention mechanism; Connect three different token-wise scalar heads at the encoder output to score the hidden state of each token: ； Among them, is the hidden vector of the t-th token, is the original score of this token, represents the projection matrix, and b is the bias term; Model the scoring difference between Chosen and Rejected in the unit test fine-tuning dataset driven by requirements based on pairwise ranking loss, and simultaneously minimize the three-way token-wise scalar head loss to obtain the optimized Critic model, including the assertion truth value error model, the answer duplication model, and the format error model.

7. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 6, wherein The pairwise ranking loss is: ； Among them, represents the overall loss of the Critic model. N is the length of the target sequence, that is, the total number of tokens, and t is the token position index. from Chosen is the original score of the Critic model for the t-th token in Chosen. from Rejected is the original score of the Critic model for the t-th token in the Rejected script. represents the Sigmoid function, which maps the difference to the interval (0, 1).

8. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 1, wherein Generate unit test cases based on the optimized Actor model, map the unit test cases to a single immediate reward based on the optimized Critic model, and obtain a large model generated based on unit test cases through reinforcement learning, including: Use the Prompt as the initial state and autoregressively generate unit test cases through the optimized Actor model; Input the unit test cases into the optimized Critic model, score each token in the complete test case, and fuse them into a single immediate reward: ； Among them, , , are the three-way scores, is the single immediate reward, , , represent the weight coefficients; Calculate the discounted cumulative return based on the single immediate reward: ； Among them, is the discounted cumulative return, t refers to the current time step or token position, where 1 ≤ t ≤ T, and T refers to the termination time step of a complete generation process. refers to the single immediate reward obtained at time step k. represents the discount factor. Calculate the advantage function based on the discounted cumulative return: ； Among them, is the scalar estimation of the optimized Critic model for the state , is the advantage function; Guide parameter updates based on the advantage function, perform policy updates on the optimized Actor model through PPO, perform value estimation updates on the optimized Critic model using A2C, and at the same time add KL penalty constraints to the policy. Iterate the above process until the target metrics converge to obtain a large model generated based on unit test cases.

9. The method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning according to claim 8, wherein The joint loss function of the large model generated based on unit test cases is: ； Among them, represents the combined loss function, represents the Policy loss, represents the value loss, represents the KL penalty constraint, represents the weight of represents the weight of; ； ； Among them, represents the probability of generating action in this round of iteration in state . represents the probability of generating action in the previous round of iteration in state . represents the importance sampling ratio, represents the advantage function, represents width, restricting policy drift, means truncating the importance sampling ratio in the interval . means averaging a batch of sampled according to the current policy at time step t; ； Among them, represents the current predicted value of the optimized Critic model, represents the predicted value of the previous iteration, represents the discounted cumulative return, represents the truncation function; ； Among them, represents the Reference Model, represents the weight coefficient, represents an asymmetric index for measuring the distance between two distributions.

Images