Large model fine-tuning method for unit test case generation based on feedback reinforcement learning

Abstract

The present invention relates to the field of software testing technology, and discloses a method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning. The method comprises the following steps: constructing a unit test fine-tuning dataset driven by requirements; constructing an Actor model, and optimizing it through the unit test fine-tuning dataset driven by requirements; constructing a Critic model, and optimizing it through the unit test fine-tuning dataset driven by requirements; generating unit test cases based on the optimized Actor model, mapping the unit test cases to a single instant reward based on the optimized Critic model, and obtaining a large model based on the unit test case generation through reinforcement learning. The present invention constructs a method for automatically generating high-quality, requirement-aligned, and consistent-style unit test cases, significantly improving test efficiency and quality, and breaking the bottlenecks of traditional test generation in terms of manual dependency, coverage depth, and requirement consistency.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more particularly to a large model fine-tuning method for generating unit test cases based on demand-driven human feedback reinforcement learning. Background Art

[0002] As the scale and iteration frequency of software systems continue to rise, unit testing has become a critical step in ensuring code quality, preventing the spread of defects, and shortening regression cycles. However, in practice, test script writing still relies primarily on manual work by development or test engineers. They must first thoroughly understand the business requirements, and then write assertions and input data based on function interfaces and potential boundary conditions. Faced with complex logic or frequently evolving requirements, this process is time-consuming and labor-intensive and easily lags behind code updates, resulting in insufficient coverage and missed regressions. Traditional automated generation technologies, such as symbolic execution, constraint solving, search testing, and static template-based unit test generation, while able to mass-produce test cases to a certain extent, often focus on syntactical validity and statement and branch coverage, with insufficient attention paid to the semantic correctness of assertions, business correctness, and code standards. The generated results still require extensive manual post-production revisions, resulting in limited overall benefits.

[0003] In recent years, large language models (LLMs) have demonstrated powerful code understanding and generation capabilities. Some research has attempted to use supervised fine-tuning (SFT) to enable large models to complete or generate unit test cases. However, the quality of test scripts in public datasets varies greatly. The "maximum likelihood" training objective cannot directly optimize metrics that can only be observed after execution, such as assertion truthfulness and path coverage. The model is prone to replicating data noise and generating code snippets that do not conform to the project style. Furthermore, test cases in industrial scenarios often must echo high-level requirements documents, and these requirements semantics cannot be captured solely through function signatures and interface annotations, resulting in a disconnect between the generated content and actual business needs.

[0004] Reinforcement learning with human feedback (RLHF) has been shown to significantly improve output quality in conversation optimization and code completion tasks. This involves first training a reward model using "chosen-rejected" examples, and then fine-tuning the large model using proximal policy optimization algorithms (such as PPO) to make its output more consistent with human preferences. However, directly applying RLHF to unit test generation presents new challenges. First, the overall reward is not available until the test script is completed, making the reward signal extremely sparse. Second, a high-quality test script must simultaneously meet multiple requirements, such as assertion correctness, path or branch coverage, and coding style standards. These goals often involve trade-offs. Third, while requirements-driven testing emphasizes that use cases must track and meet business requirements, mapping the semantics of these requirements into differentiable reward and penalty signals and continuously injecting them into the RLHF process remains an under-resolved issue.

[0005] Current research therefore presents three discontinuities: automated testing methods are difficult to accurately align with requirements, supervised fine-tuning of LLMs cannot directly optimize execution metrics, and existing RLHF applications lack fine-grained reward design and requirement constraints for testing tasks. To bridge these three gaps, a new technical framework is needed: one that fully leverages the inherent advantages of LLMs in code generation, transforms complex test quality metrics into learnable signals by decomposing reward functions, and absorbs business requirement changes and human expert feedback in real time during the training loop, so that the generated unit tests keep pace with code evolution and align with business intent. A method for fine-tuning large models for unit test case generation based on feedback reinforcement learning is proposed to break through the bottlenecks of existing automated testing in terms of labor costs, coverage depth, and requirement consistency. Summary of the Invention

[0006] In view of this, the present invention provides a large model fine-tuning method for unit test case generation based on feedback reinforcement learning. It overcomes the problem that traditional LLM supervised fine-tuning only relies on code context and cannot accurately map high-level requirements, resulting in the disconnection between generated test scripts and actual business intentions. It resolves the limitation that a single maximum likelihood training objective cannot directly optimize post-execution indicators such as assertion correctness, branch coverage, and code specifications. It eliminates the problem of unstable training of existing RLHF due to sparse rewards, difficult credit allocation, and strategy divergence in testing tasks. It ensures convergence stability through advantage estimation and KL divergence constraints; enables test cases to evolve in real time with software iterations, significantly reduces manual maintenance costs, and improves test coverage and assertion validity.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A large-scale model fine-tuning method based on demand-driven human feedback reinforcement learning unit test case generation, including:

[0009] Build a demand-driven unit test fine-tuning dataset;

[0010] Build the Actor model and fine-tune the dataset for optimization through demand-driven unit testing;

[0011] Build the critic model and optimize it by fine-tuning the dataset through demand-driven unit testing;

[0012] Unit test cases are generated based on the optimized Actor model. Unit test cases are mapped to a single immediate reward based on the optimized Critic model. A large model generated based on unit test cases is obtained through reinforcement learning.

[0013] Preferably, a demand-driven unit test fine-tuning dataset is constructed, including:

[0014] Obtain the data set and clean it;

[0015] Add a Rejected part to each training data in the dataset, assert the true value error mutation, answer duplication mutation, and format error mutation for each training data, construct the Rejected class, and obtain the demand-driven unit test fine-tuning dataset. Each data in the demand-driven unit test fine-tuning dataset is a four-tuple (Prompt, Chosen, Rejected, Response), Prompt is the prompt word, which includes the code to be tested and functional requirement information, Chosen is the result that the big model hopes to return under Prompt, Rejected is the result class that the big model does not want to return under Chosen, and Response is the actual result generated by the big model.

[0016] Preferably, the Rejected class includes a large model generated assertion true value error class, a large model generated test case duplication class, and a large model generated test case format error class.

[0017] Preferably, build an Actor model and fine-tune the dataset for optimization through demand-driven unit testing, including:

[0018] Use the OPT-1.3B large model as the backbone network and insert the LoRA layer into the attention layer;

[0019] Based on the Prompt-Chosen in the demand-driven unit test fine-tuning dataset, the OPT-1.3B large model after the above processing is supervised fine-tuned, with the NLL loss function as the target, to obtain the optimized Actor model.

[0020] Preferably, the NLL loss function for:

[0021]

[0022] in, Represents a set of model trainable parameters, including backbone Transformer weights and lightweight parameters. Indicates the conditional context, that is, through Prompt, it tells the large model what content to generate unit test cases for. Represents the true value token of the target sequence at position i, that is, the correct answer, Represents a prefix token sequence , when predicting the next token, only look at the token sequence generated before it, Representative model pair The given conditional probability is the probability of generating the current token when the model knows the prompt content and the token sequence content before the token to be generated. Indicates the target sequence length.

[0023] Preferably, build a critic model and fine-tune the dataset for optimization through demand-driven unit testing, including:

[0024] Use OPT-1.3B weights as encoder;

[0025] Freeze the backbone weights and insert the LoRA layer in the attention mechanism;

[0026] Three different token-wise scalar heads are connected to the encoder output to score the hidden state of each token:

[0027]

[0028] in, is the hidden vector of the t-th token, is the original score of the token, represents the projection matrix, b is the bias term;

[0029] Based on the pairwise ranking loss, the scoring difference between Chosen and Rejected in the requirement-driven unit test fine-tuning dataset is modeled, while minimizing the three-way token-wise scalar head loss. The optimized critic model is obtained, including the assertion truth error model, the answer duplication model, and the format error model.

[0030] Preferably, the pairwise ranking loss is:

[0031]

[0032] in, Represents the overall loss of the Critic model, N is the target sequence length, that is, the total number of tokens, t is the token position index, From Chosen , is the original score of the Critic model for the tth token in Chosen, From Rejected , is the original score of the Critic model for the tth token of the Rejected script, Represents the Sigmoid function, which maps the difference to the (0,1) interval.

[0033] Preferably, unit test cases are generated based on the optimized Actor model, and the unit test cases are mapped to a single immediate reward based on the optimized Critic model. A large model generated based on the unit test cases is obtained through reinforcement learning, including:

[0034] Use Prompt as the initial state and generate unit test cases through autoregression of the optimized Actor model;

[0035] The unit test case is input into the optimized Critic model, which scores each token in the complete test case and combines them into a single instant reward:

[0036]

[0037] in, , , For three-way scoring, For a single instant reward, 、 、 represents the weight coefficient;

[0038] Calculate the discounted cumulative reward based on a single instant reward:

[0039]

[0040] in, is the discounted cumulative return, t refers to the current time step or token position, 1 ≤ t ≤ T, T refers to the termination time step of a complete generation process, is the single immediate reward obtained at time step k, represents the discount factor;

[0041] Calculate the advantage function based on discounted cumulative return:

[0042]

[0043] in, Is the optimized Critic model for the state Scalar estimate of , is the advantage function;

[0044] The advantage function is used to guide parameter updates, and the optimized Actor model is updated with PPO. The optimized Critic model is updated with A2C using value estimation. The KL penalty constraint strategy is added and the above process is iterated until the target indicator converges to obtain a large model generated based on unit test cases.

[0045] Preferably, the large model joint loss function generated based on the unit test case is:

[0046]

[0047] in, represents the joint loss function, Indicates Policy loss, Indicates loss of value, represents the KL penalty constraint, express The weight of express The weight of

[0048]

[0049]

[0050] in, Indicates that this round of iteration is in state Produce action The probability of Indicates that in the previous iteration, the state Produce action The probability of represents the importance sampling ratio, represents the advantage function, represent Width, limiting policy drift, Indicates that the importance sampling ratio is truncated in the interval middle, Represents a batch of samples obtained by the current strategy at time step t Perform averaging;

[0051]

[0052] in, Represents the current prediction value of the optimized Critic model, represents the predicted value of the last iteration, represents the discounted cumulative return, represents the truncation function;

[0053]

[0054] in, Represents Reference Model, represents the weight coefficient, Represents an asymmetric indicator that measures the distance between two distributions.

[0055] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a method for fine-tuning a large model for generating unit test cases based on feedback reinforcement learning, which has the following effects:

[0056] 1) Significant improvement in demand consistency

[0057] A requirement parser is used to automatically convert the functional description or defect description of the module to be tested into a "requirement embedding" and implement it throughout the entire Actor-Critic reward calculation process. This ensures that the generated test script is highly consistent with the actual business requirements in terms of assertion intent, input scenarios, and boundary conditions. This allows for direct end-to-end verification of business logic and reduces regression defects caused by missing requirements.

[0058] 2) Improved test quality driven by multi-metric differentiable rewards

[0059] Dimensions such as assertion correctness, branch / path coverage, and code standardization are decomposed into differentiable rewards and integrated into an adaptively weighted total reward. Dense gradient feedback is obtained for each token, which helps the generated unit test cases approach industry-leading levels in coverage depth, assertion validity, and maintainability.

[0060] 3) More stable training convergence and high sample efficiency

[0061] The proximal policy optimization (PPO-Clip) combined with value network clipping and KL reference model constraints solves the policy divergence problem caused by sparse rewards in test scenarios; token-level advantage estimation alleviates the difficulty of credit allocation over long sequences, enabling the model to converge stably within a few rounds.

[0062] 4) Continuous iteration of parameters with high efficiency and low cost

[0063] With the LoRA lightweight module, fine-tuning can be completed by updating only <2% of the parameters; when the requirements or code baseline changes, it is only necessary to incrementally collect new requirement-test pairs and perform a small number of RLHF steps to quickly realign.

[0064] 5) Generate safe and controllable styles

[0065] The introduction of the KL penalty term aligns the supervised fine-tuning reference model to effectively suppress language drift and hallucination output; at the same time, with the help of code standard sub-rewards, the project's established code style (PEP8, pytest, etc.) is automatically maintained.

[0066] 6) Comprehensive economic and quality benefits

[0067] Deploying the solution of this invention in a continuous delivery pipeline can reduce the manual cost of writing unit tests by 60–70%, increase the early exposure rate of defects, and significantly shorten the requirement-test closed-loop time by automatically aligning with the latest requirements.

[0068] Through the above technical path, the present invention not only achieves the targeted enhancement of the unit test generation capability of large language models, but also opens up the three key links of demand-driven, quality indicator optimization and sustainable evolution, providing a new paradigm for software testing automation that is efficient, reliable and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0070] Figure 1 This is a flow chart of the method for generating large models from unit test cases based on feedback reinforcement learning provided by the present invention.

[0071] Figure 2 This is a block diagram of the method for generating large models from unit test cases based on feedback reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] The embodiment of the present invention discloses a method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning, such as Figure 1 Shown, including:

[0074] Build a demand-driven unit test fine-tuning dataset;

[0075] Build the Actor model and fine-tune the dataset for optimization through demand-driven unit testing;

[0076] Build the critic model and optimize it by fine-tuning the dataset through demand-driven unit testing;

[0077] Unit test cases are generated based on the optimized Actor model. Unit test cases are mapped to a single immediate reward based on the optimized Critic model. A large model generated based on unit test cases is obtained through reinforcement learning.

[0078] The following is a detailed description of each step:

[0079] (1) Construct a unit test fine-tuning dataset based on demand-driven testing. The dataset includes 157,767 test data for training. Each data is composed of a four-tuple (Prompt, Chosen, Rejected, Response). Prompt is a prompt word, which includes the code to be tested and functional requirement information. Chosen is the result that the large model is expected to return under Prompt. Rejected is the result class that the large model is not expected to return under Chosen. Rejected is divided into three categories, including the class of error in the true value of the assertion generated by the large model, the class of duplication of the test case generated by the large model, and the class of format error of the test case generated by the large model. Response is the actual result generated by the large model.

[0080] The specific construction process is:

[0081] Get the dataset:

[0082] You can use the "stack-dedup-python-testgen-starcoder-filter-v2" dataset on Hugging Face;

[0083] Clean the data set by removing training data longer than 6KB, training data with coverage less than 90%, and training data with test case failures.

[0084] Then, a Rejected part is added to each training data in the dataset. The dataset itself contains Prompt, Choice, and Response parts. Each training data is mutated to "assert true value error", "duplicate answer", and "format error" to construct a Rejected class and obtain a demand-driven unit test fine-tuning dataset.

[0085] (2) Build an Actor model and optimize it through demand-driven unit testing fine-tuning the dataset. Its role is to read in the prompt (tested function and requirement description) and generate a complete preliminary unit test case.

[0086] Specifically, we first selected the OPT-1.3B large model released by Facebook as the backbone. We then froze most of the original weights and inserted the LoRA layer into the attention layer, significantly reducing both video memory requirements and the number of training steps during training. Using the LoRA lightweight module, fine-tuning can be completed by updating less than 2% of the parameters. When performing supervised fine-tuning, we can use demand-driven unit testing to fine-tune the Prompt and Chosen portions of the first 40% of the training data in the dataset. Selecting the first 40% is a low-cost, low-risk, step-by-step fine-tuning strategy that ensures rapid acquisition of basic capabilities while preserving data for subsequent refined optimization and performance evaluation. Of course, we can also select training data of other numerical proportions based on actual needs. Using the NLL loss function:

[0087]

[0088] in, Represents a set of model trainable parameters, including backbone Transformer weights and lightweight parameters such as LoRA weights. Represents the conditional context, i.e., Prompt, which tells the model what content to generate unit test cases for. Represents the true value token of the target sequence at position i, that is, the correct answer. Represents a prefix token sequence , when predicting the next token, only the token sequence generated before it is looked at. Representative model pair The given conditional probability is the probability of generating the current token when the model knows the prompt content and the token sequence content before the token to be generated. It is hoped that it will be as close to 1 as possible. Represents the target sequence length. For each target token, check the probability given by the model. The lower the probability, the greater the loss, which can force the model to increase the probability of the correct token.

[0089] (3) Build a critic (scorer) model and optimize it through demand-driven unit testing fine-tuning of the dataset. Its goal is to map the unit test cases produced by the actor into scalar rewards. Use pairwise ranking loss to train multiple sub-evaluators, allowing the model to learn to score based on the semantics of Chosen and Rejected.

[0090] In the specific implementation, we use demand-driven unit testing to fine-tune the Chosen and Rejected parts of the first 20% to 60% of the training data in the dataset to obtain a more stable and generalizable ranking judgment. Of course, we can also select training data with other numerical ratios according to actual needs, and use "Chosen-Rejected" paired training models to learn what is a "good" generation result and what is a "bad" generation result, learn generation preferences, and directly reuse the same OPT-1.3B large model weights as in Actor as the encoder, freeze the backbone weights, insert the LoRA layer in the attention mechanism, and only connect three different token-wise scalar heads at the encoder output. Token-wise means token-by-token processing: the model gives a separate output for each token (word, subword or character) in the sequence. Scalar head refers to the linear output head: it converts the hidden vector of each token into a vector. (dimension d) is mapped into a :

[0091]

[0092] in, is the hidden vector of the t-th token, is the original score of the token, Represents the projection matrix (also a learnable parameter), which is responsible for transforming the hidden vector Compressed to 1 dimension, b is the bias term. Obtaining dense gradient feedback for each token helps the generated unit test cases approach industry-leading levels in coverage depth, assertion validity, and maintainability.

[0093] Based on the pairwise ranking loss, the scoring difference between Chosen and Rejected in the requirement-driven unit test fine-tuning dataset is modeled, while minimizing the three-way token-wise scalar head loss. The optimized critic model is obtained, including the assertion truth error model, the answer duplication model, and the format error model.

[0094] Pairwise ranking loss function:

[0095]

[0096] in, represents the overall loss of the Critic model. The smaller the loss, the more likely the model is to give higher Chosen scores, which is also the goal of this invention. N is the target sequence length, i.e., the total number of tokens. t is the token position index. From Chosen , refers to the original score of the Critic for the tth token in Chosen, and the higher the score, the better the quality. From Rejected , refers to the original score of the Critic for the tth token of the Rejected script. Represents the Sigmoid function, which maps the difference to the (0,1) interval. For each token, if The difference is positive, Sigmoid is close to 1, and log is close to 0. In this case, the loss contribution is small, which is encouraged. If the sigmoid is close to 0, the log is negative and has a large absolute value, the loss contribution is large, and the model will penalize this situation. The final averaging ensures that the gradients of the long and short scripts are of the same magnitude.

[0097] In the three-way token-wise scalar head of this invention, Critic-A evaluates whether assertions pass and whether execution results meet expectations (semantic correctness). Its training signals are Pass / Fail (Pass: the test case passes on the code under test, Fail: the test case fails on the code under test) labels and symbolic execution results. The evaluation score focuses on whether the assertions match the semantics of the function under test, and low scores are assigned to failed assertion tokens. Critic-B evaluates coverage (path / branch coverage). Its training signals are branch and statement coverage, and the evaluation score focuses on whether the test covers more logic. The coverage difference is used as the soft label gradient, and high coverage tokens are given positive rewards. Critic-C evaluates code standardization. For example, its training signal is the flake8 score. The evaluation score focuses on PEP8 compliance, the presence of duplicate code, and maintainability. Tokens containing standardization errors are assigned negative scores. Multi-task learning is used for training, using a backbone and three heads, while minimizing the three-way loss. Finally, adaptive weights are used for fusion and normalization.

[0098] (4) Construct the RLHF (feedback reinforcement learning) module. The function of the RLHF module is to further fine-tune the Actor model after preliminary fine-tuning through reinforcement learning to obtain a large model based on demand-driven human feedback reinforcement learning unit test case generation.

[0099] Specifically, we first perform sampling, using the Prompt (test function + requirement description) in the training set as the initial state, and generate the unit test case token sequence through the Actor model (Policy) autoregression. ;

[0100] Then, the scoring phase is carried out, and the generated unit test cases are input into the optimized Critic model, namely Critic-A, Critic-B, and Critic-C models to obtain three-way scalar scores. , , , normalized and passed through the learnable weights 、 、 Combined into a single instant reward:

[0101]

[0102] Calculating discounted cumulative returns :

[0103]

[0104] Among them, the cumulative discount return This is the total reward from time step t to the end of the episode (a complete generation process), attenuated by a discount factor. t refers to the current time step or token position, 1 ≤ t ≤ T, and T refers to the end time step of the episode. is the single immediate reward obtained at time step k. represents the discount factor, when The smaller it is, the more emphasis is placed on recent rewards; No discount. It represents the cumulative total of all rewards from time t to the end point T, and the discount factor is the component coefficient of the reward for this step. The farther away from the current location, the stronger the attenuation.

[0105] Compute the advantage function:

[0106]

[0107] in, Is the optimized Critic model for the state scalar estimate of , if It means that the action is better than the expected action, and the probability should be increased in subsequent updates, otherwise it should be reduced.

[0108] Entering the parameter update phase, PPO is used to update the policy of the optimized Actor model, and A2C is used to update the value estimation of the optimized Critic model. A2C (Advantage Actor-Critic) is a synchronous policy gradient algorithm: the Actor model gives the action selection in state s. The probability distribution of the state, the Critic network also estimates the value V(s) of the state, and the two are updated collaboratively; the advantage function is used during training Replacing the raw reward not only indicates the relative performance of the action relative to the average level, but also effectively reduces gradient variance. A KL penalty is also added to constrain the policy from deviating from the Reference Model. The Reference Model is a baseline policy frozen before entering the reinforcement learning phase. The above process is iterated until the target metrics (assertion pass rate, coverage, and format accuracy) converge.

[0109] Among them, the Actor (Policy) loss function is as follows:

[0110]

[0111]

[0112] in, Indicates that in this round of iteration (under the new strategy) in state Produce action probability. Indicates that in the previous iteration (under the old strategy) Produce action probability. stands for importance sampling ratio. represents the advantage function. represent Width, limiting policy drift, is the truncation function. Indicates that the importance sampling ratio is truncated in the interval PPO-Clip allows the policy to maximize these advantages without straying too far from the old distribution, thereby gradually optimizing towards higher rewards.

[0113] Value loss function:

[0114]

[0115] in, Represents the current prediction value of the optimized Critic model, represents the predicted value of the last iteration, represents the discounted cumulative return, where Cut for the value network.

[0116] KL penalty term:

[0117]

[0118] in Represents Reference Model, represents the weight coefficient, Represents an asymmetric indicator that measures the distance between two distributions. Here, the current strategy is measured and reference strategies The average log-probability difference between the two, with larger values ​​indicating greater differences. Its role is to prevent language hallucinations and maintain consistency with the writing style of human examples.

[0119] The joint loss function is as follows:

[0120]

[0121] In the back propagation phase, first Use your strengths as a baton to adjust your strategy, and Clip ensures updates are secure. Approach , providing a more accurate baseline for the next round. Prevent "going off topic" and ensure that the output distribution is close to human examples.

[0122] In the RLHF model, the Actor model is updated using PPO Clip, the Critic model is updated using A2C, and the Reward Model (baseline model) uses the same model as the Critic model but does not update its parameters. The Reference Model uses the same model as the Actor Model but does not update its parameters.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0124] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model fine-tuning method for generating unit test cases based on feedback reinforcement learning, characterized by: include: Build a demand-driven unit test fine-tuning dataset; Build the Actor model and fine-tune the dataset for optimization through demand-driven unit testing; Build the critic model and optimize it by fine-tuning the dataset through demand-driven unit testing; Generate unit test cases based on the optimized Actor model. Map unit test cases to single immediate rewards based on the optimized Critic model. Through reinforcement learning, a large model based on unit test case generation is obtained, including: Use Prompt as the initial state and generate unit test cases through autoregression of the optimized Actor model; The unit test case is input into the optimized Critic model, which scores each token in the complete test case and combines them into a single instant reward: r T =αR A +βR B +γR C Among them, R A , R B ,R C For three-way scoring, r T is a single immediate reward, α, β, and γ represent weight coefficients; Calculate the discounted cumulative reward based on a single instant reward: Among them, Q t is the discounted cumulative return, t refers to the current time step or token position, 1≤t≤T, T refers to the termination time step of a complete generation process, r k refers to the single immediate reward obtained at time step k, and γ represents the discount factor; Calculate the advantage function based on discounted cumulative return: δ t =Q t -V ψ (S t ) Among them, V ψ (S t ) is the optimized Critic model for state S t scalar estimate of δ t is the advantage function; The advantage function guides parameter updates, and the optimized actor model is updated with PPO. The optimized critic model is updated with A2C using value estimation. The KL penalty constraint strategy is added. The above process is iterated until the target indicator converges, resulting in a large model generated based on the unit test case. The joint loss function of the large model generated based on the unit test case is: in, represents the joint loss function, Indicates Policy loss, Indicates loss of value, represents the KL penalty constraint, λ v express The weight, λ KL express The weight of Among them, π θ (a t |s t ) indicates that this round of iteration is in state S t Produce action a t The probability of π θold (a t |s t ) indicates that in the previous iteration, the state S t Produce action a t The probability of r t (θ) represents the importance sampling ratio, δ t represents the advantage function, ε represents the clip width, limiting the strategy drift, clip(r t (θ), 1-ε, 1+ε) means that the importance sampling ratio is truncated in the interval (1-ε, 1+ε), E t Represents a batch (s) sampled according to the current strategy at time step t t , a t ) are averaged; Among them, V ψ Represents the current prediction value of the optimized Critic model, represents the predicted value of the last iteration, Q t represents the discounted cumulative return, and clip represents the truncation function; Among them, π ref represents ReferenceModel, β represents the weight coefficient, D KL Represents an asymmetric indicator that measures the distance between two distributions.

2. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 1, characterized in that: Build a demand-driven unit test fine-tuning dataset, including: Obtain the data set and clean it; Add a Rejected part to each training data in the dataset, assert the true value error mutation, answer duplication mutation, and format error mutation for each training data, construct the Rejected class, and obtain the demand-driven unit test fine-tuning dataset. Each data in the demand-driven unit test fine-tuning dataset is a four-tuple (Prompt, Chosen, Rejected, Response), Prompt is the prompt word, which includes the code to be tested and functional requirement information, Chosen is the result that the big model hopes to return under Prompt, Rejected is the result class that the big model does not want to return under Chosen, and Response is the actual result generated by the big model.

3. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 2, characterized in that: The Rejected class includes the large model generated assertion true value error class, the large model generated test case duplication class, and the large model generated test case format error class.

4. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 2, characterized in that: Build the Actor model and fine-tune the dataset for optimization through requirement-driven unit testing, including: Use the OPT-1.3B large model as the backbone network and insert the LoRA layer into the attention layer; Based on the Prompt-Chosen pair in the demand-driven unit test fine-tuning dataset, the OPT-1.3B large model after the above processing is supervised fine-tuned, and the NLL loss function is used as the target to obtain the optimized Actor model.

5. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 4 is characterized in that: NLL loss function L Actor for: Among them, θ represents the set of model trainable parameters, including the backbone Transformer weights and lightweight parameters, X represents the conditional context, that is, through Prompt, it tells the large model what content to generate unit test cases for, and y i Represents the true value token of the target sequence at position i, that is, the correct answer, y ＜i Represents a prefix token sequence (y1, y2.....y i-1 ), when predicting the next token, only the token sequence generated before is considered, pθ(y i |x,y ＜i ) represents the model for y i The given conditional probability is the probability of generating the current token when the model knows the prompt content and the token sequence content before the token to be generated. N represents the target sequence length.

6. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 2, characterized in that: Build the Critic model and fine-tune the dataset for optimization through demand-driven unit testing, including: Use OPT-1.3B weights as encoder; Freeze the backbone weights and insert the LoRA layer in the attention mechanism; Three different token-wise scalar heads are connected to the encoder output to score the hidden state of each token: h t =W proj z t +b Among them, z t is the hidden vector of the tth token, h t is the original score of the token, W proj represents the projection matrix, b is the bias term; Based on the pairwise ranking loss, the scoring difference between Chosen and Rejected in the requirement-driven unit test fine-tuning dataset is modeled, and the three-way token-wise scalarhead loss is minimized to obtain the optimized Critic model, including the assertion truth error model, the answer duplication model, and the format error model.

7. The method for fine-tuning a large model based on unit test case generation based on feedback reinforcement learning according to claim 6, characterized in that: The pairwise ranking loss is: in, Represents the overall loss of the Critic model, N is the target sequence length, that is, the total number of tokens, t is the token position index, h from Chosen t , is the original score of the Critic model for the tth token in Chosen, h from Rejected t , is the original score of the Critic model for the tth token of the Rejected script, and σ(·) represents the Sigmoid function, which maps the difference to the (0,1) interval.

Images