Method, system and equipment for improving vulnerability repairing capability of large model by utilizing multi-sample self-reward feedback mechanism and medium

Abstract

A method, system, device and medium for improving vulnerability repair ability of a large model by using a multi-sample self-reward feedback mechanism, the method comprising: combining a dynamic KL weight factor mechanism, a relative preference modeling constraint and an adaptive preference comparison mechanism of an MSR-DPO algorithm through a three-stage cascade architecture of the multi-sample self-reward feedback mechanism (MUSSEL); the technical problems that in an existing vulnerability repairing method, repairing is incomplete, accuracy is insufficient and multi-operation classification is complex are systematically solved through a unified MODIFY operation, one-time accurate vulnerability repairing based on a large language model is achieved, and an efficient and reliable technical scheme is provided for intelligent repairing in the field of code safety. The system, the equipment and the medium realize the improvement of the vulnerability repairing capability of the large model based on the method.

Description

Technical Field

[0001] This invention belongs to the field of large language model vulnerability repair technology, specifically involving a method, system, device, and medium for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism. Background Technology

[0002] The rapid development of large language model technology has led to its increasingly widespread application in programming tasks such as code generation and completion. Against this backdrop, automated software vulnerability remediation has become an important research direction, attracting continuous attention from both academia and industry. Multiple empirical studies have demonstrated the significant technological potential of large language models in this field. For example, researchers such as Pearce et al., through systematic evaluation, have confirmed that various commercial and open-source large language models can effectively remediate vulnerabilities in synthetic test cases, manually constructed vulnerabilities, and real-world security flaw scenarios.

[0003] However, existing technical solutions still have significant limitations. Unlike natural language processing tasks that can adapt to diverse outputs, code repair requires generating solutions with strict syntactic correctness and semantic accuracy. In practice, it has been found that even if the patch generated by a large language model is logically correct, minor deviations in implementation details (such as incorrectly replacing a specific variable name, resulting in the use of an undefined variable identifier) ​​will still lead to compilation failure, rendering the repair solution unusable. This technical bottleneck leads existing solutions to generally rely on a brute-force search strategy of "large-scale generation-verification": generating massive amounts of candidate patches (e.g., Pearce et al. [PEARCE, H., TAN, B., AHMAD, B., KARRI, R., AND DOLAN-GAVITT, B. Examining zero-shot vulnerability repair with large language models. In 44th IEEE Symposium on Security and Privacy, SP2023, San Francisco, CA, USA, May 21-25, 2023 (2023), IEEE, pp. 2339–2356.] generating up to 19,600 remediation solutions for 12 real-world CVEs), which are then filtered and verified by compilers and other toolchains. Ultimately, only a very small number of patches (only 982 in the aforementioned case) can be compiled and effectively repaired. This technical approach not only consumes a large amount of computing resources, but its remediation efficiency is also difficult to meet the rapid response requirements for security vulnerabilities in actual software development.

[0004] Existing vulnerability remediation technologies based on large language models can be categorized into three technical approaches based on their fine-tuning methods:

[0005] In the area of ​​supervised fine-tuning, the VRepair approach [CHEN, Z., KOMMRUSCH, S., AND MONPERRUS, M. Neural transfer learning for repairing security vulnerabilities in C code. IEEE Trans. Software Eng. 49, 1 (2023), 147–165] employs a Transformer-based model. It pre-trains the model on a large-scale general vulnerability repair dataset and then performs domain-specific fine-tuning on a specific vulnerability dataset, ultimately forming specialized vulnerability repair capabilities. The research by Zhang et al. [ZHANG, Q., FANG, C., YU, B., SUN, W., ZHANG, T., AND CHEN, Z. Pre-trained model-based automated software vulnerability repair: How far are we? CoRR abs / 2308.12533 ​​(2023).] further confirms that supervised fine-tuning can realize the transfer of large language models from general code generation to specialized vulnerability repair capabilities, providing an important theoretical foundation and practical paradigm for automated vulnerability repair technology.

[0006] In the field of reinforcement learning based on human feedback, Ouyang [OUYANG, L., WU, J., JIANG, X., ALMEIDA, D., WAINWRIGHT, CL, MISHKIN, P., AND ZHANG, C. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 (2022).] et al. utilized proximal policy optimization (PPO) [SCHULMAN, J., WOLSKI, F., DHARIWAL, P., RADFORD, A., AND KLIMOV, O. Proximal policy optimization algorithms. CoRR abs / 1707.06347] [(2017).] The algorithm aligns with the large language model, fine-tunes the policy model through supervised training, evaluates it using human feedback, and adjusts the output using the PPO algorithm.CodeDPO [ZHANG, K., LI, G., DONG, Y., XU, J., ZHANG, J., SU, J., LIU, Y., AND JIN, Z. Codedpo: Aligning code models with self-generated and verified source code. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025 (2025), Association for Computational Linguistics, pp. 15854–15871] is dedicated to improving code generation quality through external testing mechanisms. It employs bidirectional verification technology to iteratively optimize the initial self-verification scores of code snippets and test cases, thereby continuously enhancing the code generation capabilities of large language models. Focused-DPO [ZHANG, K., LI, G., LI, J., DONG, Y., AND JIN, Z. Focused-dpo: Enhancing code generation through focused preference optimization on error-prone points. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025 (2025), Association for Computational Linguistics, pp. 9578–9591] is dedicated to correcting localized errors in functional code by improving code correctness through direct preference optimization.

[0007] In the direction of reinforcement learning based on AI feedback, Bai et al.

[24] established the LLM-as-a-Judge paradigm and the rewardless model iterative DPO framework developed by Chen et al.[CHEN, Z., DENG, Y., AND YUAN, H. Self-play fine-tuning converts weaklanguage models to strong language models. CoRR abs / 2401.01335 (2024).], pioneering a new path for automated feedback. For code-large language models, existing studies have integrated RLAIF and DPO technology to optimize models such as CodeLlama-7B, confirming the significant effect of this scheme in generating code that conforms to coding standards. However, the standard DPO algorithm used in this method can only process single positive and negative sample pairs, and its identification accuracy and optimization ability are still insufficient when facing the diverse error patterns in vulnerability repair. Summary of the Invention

[0008] To overcome the problems of the existing technologies, this invention discloses a method, system, device, and medium for improving the vulnerability repair capabilities of large models using a multi-sample self-reward feedback mechanism. It constructs a multi-stage training framework by simulating the human progressive learning mechanism, establishes basic repair capabilities through supervised fine-tuning, and employs a dynamic KL-weighted direct preference optimization algorithm (MSR-DPO) based on multi-sample reinforcement learning to achieve fine-grained identification of diverse error patterns, significantly improving the accuracy of model-generated patches. Furthermore, to adapt to the generation characteristics of large language models, this invention designs a unified inference example for modification operations, simplifying multi-category code operations into a single modification instruction, effectively solving the bottleneck of operation type recognition. This technical solution maintains optimal repair performance while consuming minimal GPU memory resources; that is, it maintains excellent repair efficiency even with a very small beam size setting. Its modular design can flexibly adapt to large language models of different sizes, ensuring the practicality and deployability of the technical solution.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism, employing a three-stage cascaded architecture to achieve accurate vulnerability repair, specifically includes the following steps:

[0011] Step 1: Supervised fine-tuning of multi-sample generation stage;

[0012] 1.1 Construct a set of structured data pairs Ssft = {(x1, y1), ..., (xn, yn)} as training samples, where xi represents the original code patch containing the vulnerability, and yi represents the correctly fixed code patch annotated by the user; the training samples are constructed by collecting fix submission records from open source code platforms;

[0013] 1.2. Supervised fine-tuning of the basic large language model based on training samples is performed, and the optimal policy model is obtained by minimizing the loss function. To enhance the ability to handle C / C++ language vulnerability remediation tasks, the large language model uses the cross-entropy loss function as the optimization objective, and obtains the optimal policy model by minimizing the loss function. The cross-entropy loss function is defined as:

[0014]

[0015] in, Representing the optimal policy model The conditional probability of generating the correct patch based on the vulnerability code input under parameter configuration θ;

[0016] 1.3, Based on the obtained optimal strategy model It directly generates candidate patch sequences, including functionally correct patches, patches with syntax errors, and invalid outputs. By comparing them with verified correct patches, it constructs a complete training dataset containing preferred samples (corresponding to correct patches) and non-preferred samples (corresponding to incorrect patches).

[0017] Step 2: Self-reward feedback fine-tuning stage based on the MSR-DPO algorithm;

[0018] 2.1 Based on the complete training dataset containing preferred and unpreferred samples from step 1.3, define a dataset D containing several training instances, where each training instance in dataset D contains vulnerability code. Preference samples (i.e., the correct patch) and k non-biased samples (i.e., bug patch);

[0019] 2.2, The Direct Preference Optimization (MSR-DPO) algorithm based on multi-sample reinforcement learning is used to optimize the policy model. Make fine adjustments to differentiate the preference samples. and k non-biased samples The expected difference between them, thereby updating the policy model. The parameters are adjusted to achieve better generation performance; the loss function is defined as follows:

[0020]

[0021] in, As a reference model, For the strategy model, For activation function, It is a hyperparameter for regulating and optimizing intensity. These are dynamic unbiased weights, used to adjust the importance of multiple unbiased data points during the training process. This represents the weight parameters for supervised fine-tuning of the fusion adaptation;

[0022] 2.3, MSR-DPO algorithm's fine-grained identification of complex error modes:

[0023] 1) Dynamic KL weighting mechanism: By calculating the weighting factor for each non-preference sample... With preference samples In strategy model KL divergence under The dynamic non-preference weights are then normalized to obtain the weights. Dynamically adjust the weights of different unbiased samples during training;

[0024]

[0025] 2) Relative preference modeling constraints: For preference samples Non-preference samples By dividing by the reference model Then take the logarithm to obtain the relative probability form, i.e. and Construct relative probability indices to constrain the strategy model. Given an input, generate a conditional probability distribution for different output sequences, ensuring that it does not deviate from the initial reference model. A probability distribution benchmark to prevent reward hacking;

[0026] 3) Adaptive preference comparison mechanism: through the loss function...

[0027] Item, maximizing the preference sample Non-preference samples The relative probability difference allows the policy model to prioritize favored and non-favored sample pairs with similar probability distributions, enabling iterative learning that focuses on different error types.

[0028] Step 3: Based on the training in Steps 1 and 2, and using the unified MODIFY operation, provide inference hints to achieve vulnerability remediation.

[0029] 3.1 Design a reasoning prompt template to unify the add (ADD) and delete (DELETE) operations in code repair into a modify (MODIFY) operation;

[0030] 3.2 Introduce special positioning tags into the reasoning prompt template designed in step 3.1: using<S2SV_StartBug> and<S2SV_EndBug> Mark the location of the vulnerability in the vulnerable code, and use<S2SV_START> Mark the start position of the repair sequence using two methods.<S2SV_MODIFY> The tags limit the scope of the content to be repaired, and two...<S2SV_MODIFY> The correct repair sequence is between the tags, the first one.<S2SV_MODIFY> The tag is preceded by the context of the fragment to be repaired;

[0031] 3.3 Input the prompt containing the vulnerability code and the special location tag from step 3.2 into the fine-tuned model. The model generates a complete repair sequence through a single query, achieving one-time vulnerability repair.

[0032] The basic large language model mentioned in step 1 is an encoder-decoder model based on the Transformer architecture, and the basic large language model has built general code understanding capabilities through massive general code data in the pre-training stage; the pre-training code dataset adopts code fix submission records collected on the GitHub platform, including vulnerability type annotation, code context information and verified fix patches.

[0033] The reference model described in step 2 Compared with the optimal policy model in step 1 It is the same model, and the parameters remain fixed during the training of the MSR-DPO algorithm.

[0034] The hyperparameters mentioned in step 2 The value range of the weight parameter is 0.1-1.0. The value range is 0.05-0.3, and the optimal value is determined by grid search.

[0035] The complete repair sequence described in step 3 may contain multiple repair sub-segments, each corresponding to a vulnerability location, and each repair sub-segment follows the same principle.<S2SV_START> and<S2SV_MODIFY> Label formatting guidelines.

[0036] In step 3, the inference hint stage uses beam search to identify the tags with the highest probability in order to form a repair patch. Even with the use of beam size, MUSSEL still shows excellent efficiency and requires very little GPU memory resources.

[0037] A system for improving the vulnerability repair capabilities of large language models by utilizing a multi-sample self-reward feedback mechanism, used to implement the method described in any one of steps 1 to 3, comprising: a multi-sample generation module based on supervised fine-tuning, a self-reward feedback fine-tuning module, and a prompt query module adapted to the capabilities of large language models, wherein:

[0038] The supervised fine-tuning multi-sample generation module is used in step 1 to build structured data pairs and optimize the basic large language model with the cross-entropy loss function, thereby realizing the training of the reference model πref and the generation of candidate patch sequences containing preferred and non-preferred samples.

[0039] The self-reward feedback fine-tuning module is used in step 2. It uses the direct preference optimization algorithm (MSR-DPO) based on multi-sample reinforcement learning, combined with dynamic KL weight factor mechanism, relative preference modeling constraints and adaptive preference comparison mechanism, to achieve self-reward feedback fine-tuning of the reference model. This enables the policy model to learn the distribution characteristics of multiple preferred and non-preferred samples at the same time, thereby effectively avoiding various erroneous outputs during the vulnerability repair process and ultimately achieving one-time accurate repair.

[0040] A prompt query module adapted to the capabilities of large language models is used in step 3. This is achieved by designing a unified inference prompt template for the MODIFY operation and introducing special positioning tags.<S2SV_StartBug> ,<S2SV_EndBug> ,<S2SV_START> ,<S2SV_MODIFY> Mark the location and scope of the vulnerability, and enable one-time query-based accurate vulnerability repair of the fine-tuned model.

[0041] A device that utilizes a multi-sample self-reward feedback mechanism to improve the vulnerability repair capabilities of large models includes:

[0042] Memory, used to store computer programs;

[0043] A processor is used to improve the vulnerability repair capabilities of large models when executing the computer program according to any one of steps 1 to 3.

[0044] A computer-readable storage medium storing a computer program that, when executed by a processor, enhances the vulnerability remediation capability of large models based on the method described in any one of steps 1 to 3.

[0045] Compared with the prior art, the present invention has the following effects:

[0046] ① This invention addresses the incomplete remediation problem caused by single-sample training in existing vulnerability remediation methods. It innovatively proposes a Multi-Sample Self-Reward Feedback Mechanism (MUSSEL). By simultaneously learning the comparative features of one preferred sample and k non-preferred samples (k being a positive integer), the model can identify and avoid multiple error remediation patterns, achieving accurate remediation in a single attempt. Experimental results show that, compared to the existing state-of-the-art (SOTA) methods (BLEU score 29.32, EM score 20.00), the MUSSEL method based on the Qwen3-1.7B model achieves a BLEU score of 53.66 and an EM score of 27.66, representing an improvement of 24.34 in BLEU score and 7.66 in EM score, significantly enhancing the accuracy and completeness of vulnerability remediation.

[0047] ② This invention designs the MSR-DPO algorithm for the field of code vulnerability repair, overcoming the inherent defects of the standard DPO algorithm through three core technical mechanisms:

[0048] (1) Dynamic KL weighting factor mechanism: By calculating each non-preference sample With preference samples In strategy model KL divergence under And adopt the exponential normalization formula Dynamically assign weights and automatically adjust the importance of different unbiased samples during training.

[0049] (2) Relative preference modeling constraints: by constructing relative probability indicators By taking the logarithm, the conditional probability distribution of the sequence generated by the policy model under a given input is constrained to not deviate from the probability distribution benchmark of the reference model, effectively preventing the reward hacking problem.

[0050] (3) Adaptive preference comparison mechanism: through the loss function The algorithm maximizes the relative probability difference between preferred and unpreferred samples, causing the model to prioritize sample pairs with similar probability distributions, thus achieving iterative focused learning for different error types. By setting the hyperparameter β to a range of 0.1-1.0 and the weight parameter λ to a range of 0.05-0.3, and employing a grid search method to determine the optimal parameter combination as β=0.2 and λ=0.5, the algorithm achieves coordinated parameter tuning and optimization, significantly improving the accuracy of vulnerability repair.

[0051] ③ This invention reduces the decision complexity of the model by unifying the MODIFY operation design, abstracting the three types of operations in traditional code repair—adding (ADD), deleting (DELETE), and modifying (MODIFY)—into a single MODIFY operation. Simultaneously, it introduces four types of special location tags to construct a structured prompt template:<S2SV_StartBug> and<S2SV_EndBug> Tags are used to precisely locate the scope of vulnerable code.<S2SV_START> The tag marks the start position of the repair sequence, two<S2SV_MODIFY> The tags define the specific scope of the content to be repaired, and two...<S2SV_MODIFY> The labels represent the correct repair sequence. This design allows the fine-tuned model to generate a complete repair sequence containing multiple repair sub-fragments with a single query. Even with a small beam size, MUSSEL exhibits excellent efficiency, requiring minimal GPU memory resources to complete inference. It effectively overcomes the limitations of multi-operation classification on model generation capabilities, achieving an optimal balance between high efficiency and low resource consumption.

[0052] In summary, this invention, through a three-stage cascaded architecture of the Multi-Sample Self-Reward Feedback (MUSSEL) mechanism, combined with the dynamic KL weight factor mechanism, relative preference modeling constraints, and adaptive preference comparison mechanism of the MSR-DPO algorithm, and the structured prompt design of the unified MODIFY operation, systematically solves the technical problems of incomplete repair, insufficient accuracy, and complex classification of multiple operations in existing vulnerability repair methods. It achieves one-time accurate vulnerability repair based on a large language model, providing an efficient and reliable technical solution for intelligent repair in the field of code security. Attached Figure Description

[0053] Figure 1 This is a workflow diagram for the present invention.

[0054] Figure 2 This is an example diagram illustrating vulnerability repair using the prompt words designed according to this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings.

[0056] This invention adopts a three-stage cascaded technical architecture, which mainly includes a multi-sample generation module based on supervised fine-tuning, a self-reward feedback fine-tuning module, and a prompt query module adapted to the capabilities of large language models.

[0057] In the first stage, this invention uses a structured data pair set Ssft = {(x1, y1), ..., (xn, yn)} as training samples, where xi represents the original code patch containing the vulnerability, and yi represents the correctly patched patch annotated by humans. A domain-specific dataset is constructed by systematically collecting patch commit records from open-source code platforms (such as GitHub). Supervised fine-tuning of the basic language model is then performed to enhance its ability to handle C / C++ language vulnerability patching tasks and understand data processing design patterns, thus enabling the language model to evolve from a general dialogue tool into a specialized tool capable of specifically patching data structure defects. The optimization objective of this process is achieved by the cross-entropy loss function defined in Equation 1.

[0058] (1)

[0059] In formula (1), This represents the conditional probability that the model generates the correct patch based on the vulnerability code input under parameter configuration θ. The optimal policy model πSFT is obtained by minimizing the loss function, and a series of candidate patch sequences are generated based on this optimized model. These sequences vary greatly, ranging from perfectly functional reproductions to those plagued by syntax errors or producing absurd outputs. By comparing these sequences with verified correct patches, a complete training dataset containing both positive and negative samples is constructed. Using this data system, we can not only identify and label correct patches but also accurately locate and classify erroneous patches, thus providing comprehensive data support for subsequent training stages by establishing multi-dimensional quality samples.

[0060] In the second phase, to understand the self-reward feedback fine-tuning mechanism of this invention, it is necessary to first explain the common practices of traditional models. Traditional methods employ reinforcement learning based on human feedback after supervised fine-tuning to ensure output quality. The core process includes: the annotator selecting a preferred sample y from multiple candidate outputs. w With unbiased sample y l (In this invention, these correspond to correct and incorrect patches, respectively), based on the triplet dataset P=(x, y) w , y l ), reference model π ref (Typically, this refers to large language models with fixed parameters) and supervised fine-tuning of generative models π. θ The policy model parameters are optimized using the direct preference optimization algorithm (as shown in Equation (2)):

[0061] (2)

[0062] The optimization objective is to align the output probability distribution of the generated sequences with the preferred data while moving away from the unpreferred data. The reference model effectively avoids deviations from the expected distribution, such as generating sequences with abnormal lengths or repeating sequences, by calculating the relative distance between the conditional probabilities of the preferred and unpreferred data.

[0063] However, the standard DPO algorithm has limitations in vulnerability remediation tasks: each training instance only supports pairing one non-preference sample with a single preference sample, while vulnerability remediation requires a large number of non-ideal outputs to construct a non-preference dataset. Therefore, this invention proposes the MSR-DPO algorithm (Direct Preference Optimization Algorithm Based on Multi-Sample Reinforcement Learning), which specifically adjusts and enhances the DPO algorithm for vulnerability remediation tasks. Its loss function is defined as shown in formula (3):

[0064]

[0065] Weight Defined by formula (4):

[0066] Indicates from the generative model π θ Multiple unbiased samples from the middle, y w This represents the expected correct samples sampled concurrently. In this algorithm framework, σ is the activation function, β is the hyperparameter that regulates the optimization intensity, γi is used as a dynamic non-preference weight to adjust the importance of multiple non-preference data during the training process, and λ represents the weight parameter of supervised fine-tuning fusion adaptation. The first term of formula (3) calculates the relative distance between the policy model and the reference model on the probability distribution of the output of the preference data (i.e., the relative reference modeling mechanism, whose principle is consistent with the standard DPO algorithm); the second term calculates the relative distance of the output of multiple non-preference data (i.e., the preference contrast loss mechanism); the last term introduces the supervised fine-tuning fusion adaptation loss, which also helps to prevent overfitting caused by preference optimization. The core objective of this algorithm is to optimize the expected difference between the correct generated results and multiple incorrect generated results, and to achieve a systematic improvement in generation performance by updating the policy model parameters.

[0067] Formula (3) has three core features that need to be emphasized:

[0068] First, the dynamic KL weighting mechanism. During the preference optimization training phase, the model simultaneously receives one desired preferred sample and multiple non-preferred samples. As training progresses, the probability distribution of the model-generated sequences and the differences between them and the multiple non-preferred samples continuously evolve. The weighting factor γ in the formula... i Through molecular part Continuously calculate the KL divergence between the i-th unfavorable sample and the favorable sample, and then calculate it through the denominator. Normalization is performed to dynamically adjust the importance of the loss associated with different unbiased samples, enabling the model to make full use of these unbiased samples and dynamically identify differences during training to achieve optimal learning results.

[0069] Second, relative preference modeling constraints. The KL divergence calculation in the formula is based on the relative preference modeling method. This method prevents reward hacking by focusing on the relative probabilities of different outcomes rather than just the reward model within the policy framework. This prevents the model from unexpectedly exploiting the reward system to maximize its reward without actually completing the intended task. Specifically, for preference samples... Non-preference samples By dividing by the reference model Then take the logarithm to obtain the relative probability form. and The relative probability is used to constrain the policy optimization of the policy model using a reference model, preventing it from generating sequences whose probability distribution deviates significantly from the desired distribution.

[0070] Third, the adaptive preference contrast mechanism. The preference contrast loss guides the policy model π by maximizing the relative difference in preference values ​​between preferred and unpreferred data. θ Optimize in the desired direction. Different loss magnitudes provide dynamic weights for training, allowing the model to repeatedly examine a "set of incorrect cases," achieving targeted learning. During training, π θ The model dynamically adapts to changing reward margins while maximizing relative reward differences. It calculates the probability distribution difference between preferred and unpreferred outputs, prioritizing sample pairs with similar probability distributions. This results in varying attention to unpreferred samples across different training epochs, forming an iterative learning pattern that focuses on different error types. This allows for the simultaneous handling of multiple unpreferred objectives in a single optimization step, which is crucial for complex tasks requiring parallel avoidance of various undesirable outputs, significantly improving training efficiency and effectiveness.

[0071] In the third stage, this invention improves the data preprocessing strategy to better adapt to large language models. We simplify the repair mode, retaining only the MODIFY operation and discarding the three different operation types: ADD, DELETE, and MODIFY. (See attached...) Figure 2Let's take an example to illustrate our method: In the "original vulnerability code", we use<S2SV_StartBug> and<S2SV_EndBug> Label the location of the vulnerability. In the "Output" section,<S2SV_START> Mark the starting position of the repair.<S2SV_MODIFY> The content between the tags contains the context of the repair sequence, typically including several markers preceding the fragment to be repaired. The correct repair sequence is located between two tags.<S2SV_MODIFY> Between tags. The model's task is to generate a complete fix sequence for the entire method through a single query. This sequence typically contains multiple fix sub-segments. This carefully designed hint format, which includes the vulnerability code and its fix, is also applied during the model training phase.

[0072] Simulation Experiment

[0073] I. Experimental Environment Configuration

[0074] This invention uses the CVEFixes and Big-Vul datasets for validation experiments. These datasets contain 5,800 C / C++ vulnerability records from 1,754 real open-source projects, covering over 180 CWE (Common Weakness Enumeration) types. The datasets are divided into 3,872 training samples, 316 validation samples, and 1,612 test samples to ensure no data leakage between the training, validation, and test sets.

[0075] The hardware environment for the experiment consisted of two NVIDIA A100-80G graphics processors and four RTX 3090-24G graphics processors; the software environment used the LlamaFactory framework, combined with the PyTorch deep learning library and the Transformers model library.

[0076] The experimental parameters are set as follows:

[0077] (1) In the supervised fine-tuning stage of step S1, LoRA (Low-Rank Adaptation) adapter is used for efficient parameter fine-tuning, and the learning rate is set to 5×10. -5 Lora rank is set to 10, Lora target is set to "all", and the training rounds exceed 8 epochs;

[0078] (2) In the multi-sample self-reward preference optimization stage of step S2, full parameter adapter fine-tuning is performed, the Beta factor is set to 0.2, the Ftx factor is set to 0.5, and the training rounds exceed 14 epochs; FP16 precision training is used to reduce GPU memory usage.

[0079] II. Performance Comparison Experiment

[0080] This invention uses BLEU score and EM score as the main evaluation indicators to assess the alignment degree between the generated repair sequence and the target repair sequence. The BLEU score measures sequence similarity, while the EM score measures the perfect match rate.

[0081]

[0082] Table 1: Performance Comparison of Different Methods

[0083] The experimental results are shown in Table 1. General pre-trained models (such as CodeBERT, PolyCoder, and CodeT5) perform poorly without task-specific fine-tuning, with BLEU scores of only 3.92-4.35 and EM scores of 3.52-10.23. Regarding task-specific models, VRepair achieves a BLEU score of 24.29 and an EM score of 8.91; the current best method, VulMaster, achieves a BLEU score of 29.32 and an EM score of 20.00. These methods are limited by their single-sample learning mechanism, and there is still room for improvement in both the completeness and accuracy of vulnerability repair. Although closed-source large language models (GPT-3.5 and GPT-4o) possess powerful general language understanding capabilities, their vulnerability repair performance (BLEU scores of 8.81-9.74 and EM scores of 3.64-5.31) is significantly lower than that of specially trained models without domain-specific fine-tuning.

[0084] In contrast, supervised fine-tuning models showed significant improvements: StableCode-3B achieved a BLEU score of 35.82 and an EM score of 4.21; CodeGemma-3B achieved a BLEU score of 38.87 and an EM score of 6.93; DeepSeek-Coder-1.3B achieved a BLEU score of 47.12 and an EM score of 16.09; DeepSeek-R1-1.5B achieved a BLEU score of 48.91 and an EM score of 19.66; and Qwen3-1.7B achieved a BLEU score of 48.18 and an EM score of 20.16. However, these models only employed the traditional SFT method and failed to fully utilize information from incorrect samples for comparative learning.

[0085] The proposed MUSSEL method achieves optimal performance: the MUSSEL method based on DeepSeek-Coder-1.3B achieves a BLEU score of 51.31 and an EM score of 24.25; the MUSSEL method based on DeepSeek-R1-1.5B achieves a BLEU score of 53.19 and an EM score of 26.67; and the MUSSEL method based on Qwen3-1.7B achieves a BLEU score of 53.66 and an EM score of 27.66. Compared with the existing state-of-the-art method VulMaster, the MUSSEL method based on Qwen3-1.7B improves the BLEU score by 24.34 points (a relative improvement of 83.01%) and the EM score by 7.66 points (a relative improvement of 38.30%), fully validating the effectiveness of the multi-sample self-reward feedback mechanism and the MSR-DPO algorithm.

[0086] In summary, this invention achieves superior performance on the CVEFixes and Big-Vul standard datasets by combining a multi-sample self-reward feedback mechanism (MUSSEL) with the MSR-DPO algorithm. The MUSSEL method based on the Qwen3-1.7B model achieves excellent BLEU scores of 53.66 and EM scores of 27.66. Compared with the existing state-of-the-art method VulMaster, this invention achieves a significant improvement of 83.01% in BLEU score and 38.30% in EM score accuracy, fully verifying the advanced technology and superior performance of this method in automatic vulnerability repair tasks, and providing an efficient and reliable technical solution for intelligent code security repair.

Claims

1. A method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism, characterized in that, A three-stage cascaded architecture is used to achieve precise vulnerability remediation, specifically including the following steps: Step 1: Supervised fine-tuning of multi-sample generation stage; 1.1 Construct a set of structured data pairs Ssft = {(x1, y1), ..., (xn, yn)} as training samples, where xi represents the original code patch containing the vulnerability, and yi represents the correctly fixed code patch annotated by the user; the training samples are constructed by collecting fix submission records from open source code platforms; 1.

2. Supervised fine-tuning of the basic large language model based on training samples is performed, and the optimal policy model is obtained by minimizing the loss function. To enhance the ability to handle C / C++ language vulnerability remediation tasks, the large language model uses the cross-entropy loss function as the optimization objective, and obtains the optimal policy model by minimizing the loss function. The cross-entropy loss function is defined as: in, Representing the optimal policy model The conditional probability of generating the correct patch based on the vulnerability code input under parameter configuration θ; 1.3, Based on the obtained optimal strategy model It directly generates candidate patch sequences, including functionally correct patches, patches with syntax errors, and invalid outputs. By comparing them with verified correct patches, it constructs a complete training dataset containing preferred samples (corresponding to correct patches) and non-preferred samples (corresponding to incorrect patches). Step 2: Self-reward feedback fine-tuning stage based on the MSR-DPO algorithm; 2.1 Based on the complete training dataset containing preferred and unpreferred samples from step 1.3, define a dataset D containing several training instances, where each training instance in dataset D contains vulnerability code. Preference samples (i.e., the correct patch) and k non-biased samples (i.e., bug patch); 2.2, The Direct Preference Optimization (MSR-DPO) algorithm based on multi-sample reinforcement learning is used to optimize the policy model. Make fine adjustments to differentiate the preference samples. and k non-biased samples The expected difference between them, thereby updating the policy model. The parameters are adjusted to achieve better generation performance; the loss function is defined as follows: in, As a reference model, For the strategy model, For activation function, It is a hyperparameter for regulating and optimizing intensity. These are dynamic unbiased weights, used to adjust the importance of multiple unbiased data points during the training process. This represents the weight parameters for supervised fine-tuning of the fusion adaptation; 2.3, MSR-DPO algorithm's fine-grained identification of complex error modes: 1) Dynamic KL weighting mechanism: By calculating the weighting factor for each non-preference sample... With preference samples In strategy model KL divergence under The dynamic non-preference weights are then normalized to obtain the weights. Dynamically adjust the weights of different unbiased samples during training; 2) Relative preference modeling constraints: For preference samples Non-preference samples By dividing by the reference model Then take the logarithm to obtain the relative probability form, i.e. and Construct relative probability indices to constrain the strategy model. Given an input, generate a conditional probability distribution for different output sequences, ensuring that it does not deviate from the initial reference model. A probability distribution benchmark to prevent reward hacking; 3) Adaptive preference comparison mechanism: through the loss function... Item, maximizing the preference sample Non-preference samples The relative probability difference allows the policy model to prioritize favored and non-favored sample pairs with similar probability distributions, enabling iterative learning that focuses on different error types. Step 3: Based on the training in Steps 1 and 2, and using the unified MODIFY operation, provide inference hints to achieve vulnerability remediation. 3.1 Design a reasoning prompt template to unify the add (ADD) and delete (DELETE) operations in code repair into a modify (MODIFY) operation; 3.2 Introduce special positioning tags into the reasoning prompt template designed in step 3.1: using<S2SV_StartBug> and<S2SV_EndBug> Mark the location of the vulnerability in the vulnerable code, and use<S2SV_START> Mark the start position of the repair sequence using two methods.<S2SV_MODIFY> The tags limit the scope of the content to be repaired, and two...<S2SV_MODIFY> The correct repair sequence is between the tags, the first one.<S2SV_MODIFY> The tag is preceded by the context of the fragment to be repaired; 3.3 Input the prompt containing the vulnerability code and the special location tag from step 3.2 into the fine-tuned model. The model generates a complete repair sequence through a single query, achieving one-time vulnerability repair.

2. The method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism according to claim 1, characterized in that, The basic large language model mentioned in step 1 is an encoder-decoder model based on the Transformer architecture, and the basic large language model has built general code understanding capabilities through massive general code data in the pre-training stage; the pre-training code dataset adopts code fix submission records collected on the GitHub platform, including vulnerability type annotation, code context information and verified fix patches.

3. The method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism according to claim 1, characterized in that, The reference model described in step 2 Compared with the optimal policy model in step 1 It is the same model, and the parameters remain fixed during the training of the MSR-DPO algorithm.

4. The method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism according to claim 1, characterized in that, The hyperparameters mentioned in step 2 The value range of the weight parameter is 0.1-1.

0. The value range is 0.05-0.3, and the optimal value is determined by grid search.

5. The method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism according to claim 1, characterized in that, The complete repair sequence described in step 3 may contain multiple repair sub-segments, each corresponding to a vulnerability location, and each repair sub-segment follows the same principle.<S2SV_START> The formatting specifications for the <S2SV_MODIFY> tag.

6. The method for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism according to claim 1, characterized in that, In step 3, the inference hint stage uses beam search to identify the tags with the highest probability in order to form a repair patch. Even with the use of beam size, MUSSEL still shows excellent efficiency and requires very little GPU memory resources.

7. A system for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism, used to implement the method described in any one of claims 1 to 6, characterized in that, include: The system comprises a supervised fine-tuning multi-sample generation module, a self-reward feedback fine-tuning module, and a prompt query module adapted to the capabilities of large language models, wherein: The supervised fine-tuning multi-sample generation module is used in step 1 to build structured data pairs and optimize the basic large language model with the cross-entropy loss function, thereby realizing the training of the reference model πref and the generation of candidate patch sequences containing preferred and non-preferred samples. The self-reward feedback fine-tuning module is used in step 2. It uses the direct preference optimization algorithm (MSR-DPO) based on multi-sample reinforcement learning, combined with dynamic KL weight factor mechanism, relative preference modeling constraints and adaptive preference comparison mechanism, to achieve self-reward feedback fine-tuning of the reference model. This enables the policy model to learn the distribution characteristics of multiple preferred and non-preferred samples at the same time, thereby effectively avoiding various erroneous outputs during the vulnerability repair process and ultimately achieving one-time accurate repair. A prompt query module adapted to the capabilities of large language models is used in step 3. This is achieved by designing a unified inference prompt template for the MODIFY operation and introducing special positioning tags.<S2SV_StartBug> ,<S2SV_EndBug> ,<S2SV_START> ,<S2SV_MODIFY> Mark the location and scope of the vulnerability, and enable one-time query-based accurate vulnerability repair of the fine-tuned model.

8. A device for improving the vulnerability repair capability of large models by utilizing a multi-sample self-reward feedback mechanism, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program, enhance the vulnerability repair capability of large models according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can improve the vulnerability repair capability of large models based on the method described in any one of claims 1 to 6.

Images