A Software Vulnerability Detection and Inference Method Based on Reward-Driven Large Language Model
By employing a reward-driven large language model approach, utilizing a two-stage teacher model and reinforcement learning techniques, we generate inference and explanation texts that are consistent with real code dependencies. This addresses the issues of insufficient detection performance and inadequate automated quality of vulnerability inference and explanation in existing software vulnerability detection methods, thereby improving the accuracy and interpretability of vulnerability detection.
Patent Information
- Application Number
- CN202611320035.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing software vulnerability detection methods based on large language models suffer from limited vulnerability detection performance, insufficient generalization ability, inadequate handling of false positives and false negatives, lack of automated quality supervision mechanisms for vulnerability reasoning and interpretation, and scarcity of high-quality vulnerability reasoning and interpretation data.
We employ a reward-driven large language model approach, constructing a training dataset through a two-stage teacher model generation and validation framework. We then combine result-supervised rewards, process structure-aware rewards, and overall semantic consistency rewards for reinforcement learning to generate inference and explanation texts that are consistent with real code dependencies and vulnerability triggering paths.
It improves the accuracy and interpretability of vulnerability detection, reduces the risk of false negatives, generates high-quality reasoning and explanation texts, supports security personnel in directly auditing the causes of vulnerabilities and locating the root causes of problems, and simplifies the deployment process.
Smart Images

Figure CN123116288A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of software security and artificial intelligence technology, specifically relating to a software vulnerability detection and reasoning interpretation method based on a reward-driven large language model. Background Technology
[0002] Software vulnerabilities are security weaknesses introduced by errors in program design, coding implementation, or resource management. Attackers can exploit these vulnerabilities to cause security risks such as unauthorized access, privilege escalation, denial of service, information leakage, or system damage. With the continuous expansion of open-source software, cloud services, mobile terminals, and critical infrastructure software, source code vulnerability detection has become a crucial aspect of software security assurance.
[0003] Existing vulnerability detection technologies mainly include static analysis methods, dynamic analysis methods, deep learning methods, and methods based on large language models.
[0004] Static analysis methods typically rely on manually defined rules, pattern matching, symbolic execution, or program path analysis to scan source code and discover potential vulnerabilities. These methods offer advantages such as high interpretability and the ability to run the program, but they are highly dependent on manually defined rules and the capabilities of the analyzer. When the codebase is large, the call relationships are complex, or the vulnerability patterns are unclear, static analysis methods are prone to high false positive rates, insufficient rule coverage, and limited scalability.
[0005] Dynamic analysis methods typically discover anomalous behavior during program execution through fuzz testing, taint analysis, or runtime monitoring. These methods can verify whether a vulnerability can be triggered, but they require constructing valid test inputs and covering the vulnerability path. In complex software, vulnerability trigger paths are often deep and sparse, and dynamic analysis methods may consume significant computational resources and struggle to provide a complete root cause explanation for each detection result.
[0006] Deep learning methods learn the semantic and structural features of code through neural networks to achieve end-to-end vulnerability detection. Existing research uses code representations such as abstract syntax trees, control flow graphs, program dependency graphs, code attribute graphs, and code sequences. While these methods improve the automation of vulnerability detection, they still have the following shortcomings: First, model performance depends on the distribution of training data, and its generalization ability is limited when crossing projects, coding styles, or vulnerability types; second, models typically only output vulnerability labels or probability scores, making it difficult to explain why the code is judged to be vulnerable; third, the prediction results given by the models lack an auditable reasoning process, requiring security personnel to conduct extensive manual analysis to confirm the causes and impact paths of vulnerabilities.
[0007] In recent years, large language models have been introduced into vulnerability detection tasks. One approach involves inputting source code and security knowledge into the large language model through prompting engineering, allowing the model to directly determine the existence of vulnerabilities. Another approach uses supervised fine-tuning to enable the large language model to learn from vulnerability detection samples and labeled answers. Compared to traditional deep learning models, large language models possess stronger capabilities in natural language understanding, code comprehension, and interpretation generation, and can generate analytical explanations along with the output of detection results.
[0008] However, existing vulnerability detection methods based on large language models still have the following problems. First, relying solely on hint engineering is insufficient to fully model the deep semantics and execution paths of vulnerable code. Even if vulnerability type, control flow, or dependency information is added to the hints, the model may only superficially reference this information without truly making a judgment based on the program's semantics. Second, supervised fine-tuning can easily enable the model to learn fixed output formats or shallow vulnerability patterns, but it may not be able to learn reliable vulnerability reasoning processes. The model may generate incorrect, repetitive, generalized, or hallucinatory interpretations even if the final classification result is correct. Third, existing methods do not adequately handle the costs of false positives and false negatives. In security scenarios, false negatives may lead to the neglect of real vulnerabilities, and their risk is usually higher than that of false positives; the traditional training or evaluation method of "giving points for correct judgments and no points for incorrect judgments" cannot reflect this asymmetric risk. Fourth, vulnerability reasoning explanation texts lack automated quality supervision mechanisms. Mathematical reasoning tasks can usually be automatically verified through the final answer or intermediate calculation results, but vulnerability reasoning explanations are in natural language form, containing factors such as code semantics, variable dependencies, control paths, and security semantics, making it difficult to directly judge whether each step is correct. Fifth, high-quality vulnerability reasoning data is scarce. Publicly available vulnerability datasets typically provide source code, vulnerability tags, vulnerability lines, patch submissions, or vulnerability descriptions, but rarely offer step-by-step inference explanation text for model training. Therefore, how to construct large-scale, high-quality vulnerability inference explanation data that can be used for reinforcement fine-tuning is a fundamental problem in applying large language models to explainable vulnerability detection. Summary of the Invention
[0009] To address the aforementioned issues, this invention discloses a software vulnerability detection and reasoning explanation method based on a reward-driven large language model. This method not only improves the performance of binary vulnerability detection but also generates reasoning explanation text that is consistent with the actual code dependencies and vulnerability triggering paths, thereby enhancing the accuracy, interpretability, and engineering usability of vulnerability detection results.
[0010] To achieve the above objectives, the present invention adopts the following technical solution:
[0011] A software vulnerability detection and reasoning explanation method based on a reward-driven large language model includes the following steps:
[0012] S1, acquire function-level vulnerability sample data, and perform code semantic filtering, function length constraint and semantic integrity check on the sample data to obtain preprocessed sample data;
[0013] S2, using a two-stage teacher model generation and verification framework, automatically generates a training dataset containing stepwise reasoning explanation text and vulnerability labels for the preprocessed sample data;
[0014] S3, Use the training dataset to perform supervised fine-tuning of the basic large language model, so that the model learns the mapping relationship from source code to inference explanation text and vulnerability labels, and obtains the initial policy model;
[0015] S4, starting with the initial policy model, an output sequence is generated through online sampling, and the model is trained using reinforcement learning based on a composite reward function to obtain a reinforcement fine-tuning model; wherein, the composite reward function includes at least a result supervision reward for evaluating the correctness of the vulnerability binary classification result. The process structure-aware reward used to evaluate the consistency between the reasoning explanation text and the actual dependency path of the code in the output sequence. And a global semantic consistency reward used to evaluate the global semantic consistency between the inference explanation text and the reference inference explanation text in the output sequence. ;
[0016] S5, input the source code to be detected into the enhanced fine-tuning model, and the model outputs the detection result containing the stepwise reasoning explanation text, vulnerability description and binary vulnerability prediction label in a single forward inference.
[0017] Further, step S2 includes: inputting source code, vulnerability labels, and structured hints into the first teacher model to generate a structured result containing the step-by-step reasoning process, and performing format checks and conclusion consistency self-checks on the generated result, triggering regeneration for results that do not meet the requirements; inputting the reasoning explanation text that has passed the self-check into the second teacher model to perform semantic consistency audit and security semantic matching audit; wherein, the semantic consistency audit is used to check whether the reasoning steps correspond to specific code statements and whether there are fictitious variables or functions, and the security semantic matching audit is used to check whether the reasoning explanation conforms to the vulnerability category and vulnerability description and does not change the original vulnerability label; the reasoning explanation text that has passed the dual audits, together with the corresponding source code and vulnerability labels, constitutes the training dataset.
[0018] Furthermore, in step S4, the result is supervised and rewarded. The calculation method is as follows: the vulnerability labels predicted by the model are compared with the actual labels. Based on four scenarios, namely, the predicted vulnerability is a vulnerability and it is a vulnerability, the predicted non-vulnerability is a non-vulnerability, the predicted vulnerability is a vulnerability and it is a non-vulnerability, and the predicted non-vulnerability is a vulnerability and it is a vulnerability, a first positive reward, a second positive reward, a first negative reward, and a second negative reward are assigned respectively. The first positive reward is greater than the second positive reward, and the penalty intensity of the second negative reward is greater than the penalty intensity of the first negative reward.
[0019] Furthermore, in step S4, process structure perception reward The calculation method is as follows: For a vulnerability sample, a code attribute graph is constructed for the function, and a backward program slice is executed starting from the vulnerability behavior. Variable names and function names in the vulnerability statement are extracted to form a zero-level keyword set. Variable names and function names in the first-level dependency statement are extracted to form a first-level keyword set. Variable names and function names in the second-level dependency statement are extracted to form a second-level keyword set, and decreasing layer weights are assigned to each level. Variable names, function names, and code-related keywords in each inference step generated by the model are extracted to form a step keyword set. The frequency overlap between the step keyword set and the keyword sets of each level is calculated, and the results are weighted and summed according to the layer weights to obtain the step reward for that inference step. The average of the step rewards for all inference steps is taken to obtain the process structure perception reward.
[0020] Furthermore, process structure perception reward Represented as:
[0021] ,
[0022] ,
[0023] ,
[0024] The generated reasoning content contains N reasoning steps. The reward for reasoning in step n. Keywords The number of times it appears in set X This indicates that all keywords are traversed. Used to count the number of valid matches between the current inference step and the corresponding dependency layer; The unit reward value for the keyword at the k-th level. Indicates the total reward scale. and These represent the number of keyword types and the total number of occurrences at the k-th level, respectively.
[0025] Furthermore, for non-vulnerable samples, representative variable names and function names are extracted based on the program structure information in the code attribute graph to form a non-vulnerable structure keyword set. The inference steps generated by the computation model overlap with the keyword frequency of the non-vulnerable structure keyword set to obtain the process structure perception reward of the non-vulnerable sample.
[0026] Furthermore, in step S4, the overall semantic consistency reward The calculation method is as follows: the reasoning explanation text generated by the model is denoted as the generated explanation, and the reasoning explanation text generated by the teacher model is denoted as the reference explanation; the semantic vector representations of the generated explanation and the reference explanation are obtained by using a sentence embedding model; the generated explanation and the reference explanation are divided into multiple aligned text segments by using a sliding window semantic aggregation strategy, and the cosine similarity between the semantic vectors of each group of aligned segments is calculated; the average of the cosine similarity of all aligned segments is taken to obtain the overall semantic consistency reward.
[0027] Furthermore, the sentence embedding model is all-mpnet-base-v2.
[0028] Furthermore, overall semantic consistency reward Represented as: Where E(⋅) represents the sentence encoding function, and Let i represent the i-th predicted text and the i-th reference text, respectively.
[0029] Furthermore, the composite reward function is expressed as: ,in, To reward the balance coefficient, The strategy learned so far, Let β be the initial reference strategy and β be the KL divergence penalty coefficient.
[0030] Furthermore, in step S3, the model parameters are updated using a proximal strategy optimization algorithm, and the parameters are finely tuned efficiently using a low-rank adaptation method, updating only the low-rank adaptation parameters instead of all model parameters.
[0031] Furthermore, in step S4, the reasoning and interpretation text analyzes key variables, key functions, input boundaries, memory operations, resource management, control flow conditions, and data dependencies step by step according to the actual execution order of the functions. If a vulnerability is found, the vulnerable statement, vulnerability type, triggering condition, and impact are indicated. If no vulnerability is found, the key security checks, boundary constraints, error handling, or resource management logic are explained.
[0032] Beneficial effects:
[0033] (1) This invention uses asymmetric result supervision and reward to impose strong penalties on false negatives and give high rewards for correctly detecting vulnerabilities, thereby guiding the model to learn the risk preferences of security scenarios, effectively reducing the risk of false negatives and improving the ability to detect real vulnerabilities;
[0034] (2) This invention guides the model to pay attention to the real variables and functions in the code attribute graph and program dependency path when generating inference steps through process structure perception reward, so that the inference interpretation is consistent with the actual execution logic of the source code, reducing the problems of illusion and path inconsistency;
[0035] (3) This invention uses overall semantic consistency reward to constrain the consistency between the generated explanation and the reference explanation from the global semantic level, prevents the model from obtaining high scores by stuffing keywords, and ensures the logical coherence and semantic correctness of the reasoning explanation;
[0036] (4) This invention automatically constructs a training dataset containing step-by-step reasoning explanations through a two-stage teacher model generation and verification framework, freeing it from the dependence on manual annotation and obtaining large-scale, highly reliable training data at a controllable cost.
[0037] (5) During the inference stage of this invention, the progressive inference explanation text, vulnerability description and binary vulnerability prediction label are output simultaneously, supporting security personnel to directly audit the cause of the vulnerability and locate the root cause of the problem; detection can be completed in a single forward inference, without the need to call external tools, which is convenient for integration and deployment;
[0038] (6) The present invention uses the low-rank adaptation (LoRA) method to efficiently fine-tune parameters, only updating a small number of adaptation parameters, reducing training memory overhead; only a single reinforcement fine-tuning model is needed in the inference stage, without the need to attach a teacher model or external analysis tools, simplifying the deployment architecture. Attached Figure Description
[0039] Figure 1 This is a general framework diagram of an embodiment of the present invention;
[0040] Figure 2 This is a flowchart illustrating the multi-dimensional reward enhancement fine-tuning process according to an embodiment of the present invention.
[0041] Figure 3 This is a schematic diagram of the process structure perception reward calculation according to an embodiment of the present invention;
[0042] Figure 4 This is an example of the CVE-2015-4003 vulnerability function oz_cdev_write in an embodiment of the present invention. Detailed Implementation
[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0044] Existing software vulnerability detection methods generally suffer from limited detection performance, insufficient generalization ability, and a lack of credible vulnerability explanations. Traditional methods either only output a binary label or score without explaining the basis for the judgment, or while they may have explanation capabilities, the explanations often deviate from the actual semantics of the code, exhibiting issues such as illusions and inconsistent paths, requiring security personnel to perform extensive manual analysis to confirm the cause of the vulnerability. This invention addresses function-level source code vulnerability detection tasks by constructing a vulnerability detection dataset with inference explanation text. It sequentially performs supervised fine-tuning and reinforcement fine-tuning on a large language model, enabling the model to simultaneously output vulnerability inference explanation text, vulnerability description, and binary vulnerability prediction results after inputting the source code to be detected. This invention can be applied to automated auxiliary analysis scenarios in software security detection, source code security auditing, vulnerability-assisted localization, vulnerability cause explanation, code security assessment, and secure development processes.
[0045] Example 1
[0046] This invention proposes a software vulnerability detection and reasoning explanation method and system based on a reward-driven large language model. The core design idea of this application is: first, to construct training data with reasoning explanation text through a two-stage teacher model generation and verification framework; then, to obtain a stable initial policy model through supervised fine-tuning; and finally, to drive reinforcement fine-tuning through result-supervised rewards, process structure-aware rewards, and overall semantic consistency rewards, so that the model can generate high-quality reasoning explanations that are more consistent with code structure and vulnerability semantics while improving detection accuracy.
[0047] like Figure 1 As shown, the method takes function-level vulnerability samples as input and goes through six stages to finally obtain a reinforcement fine-tuning model that can simultaneously output vulnerability reasoning explanations and binary classification vulnerability prediction results. The six stages are: data preprocessing, which performs semantic filtering, length constraints, and integrity checks on function-level vulnerability samples; reasoning explanation text generation and verification, which uses a two-stage teacher model to construct a training dataset containing progressive reasoning explanations and vulnerability labels; supervised fine-tuning warm-up, which enables the basic large language model to learn the mapping from source code to reasoning explanations and labels to obtain an initial policy model; reinforcement fine-tuning, which uses the initial policy model as a starting point for online sampling and updates the model parameters based on a composite reward function consisting of result supervision rewards, process structure awareness rewards, and overall semantic consistency rewards through a proximal policy optimization algorithm; and vulnerability detection reasoning, which inputs the source code to be detected into the reinforcement fine-tuning model and outputs reasoning explanation text, vulnerability description, and binary classification vulnerability prediction labels in a single forward inference.
[0048] The specific steps are as follows:
[0049] (1) Data preprocessing
[0050] The system first acquires function-level vulnerability sample data. This function-level vulnerability sample data includes at least one or more of the following: function source code, vulnerability tags, vulnerability line information, vulnerability description, vulnerability category, and patch submission information. In one embodiment, the source data may be a publicly available function-level vulnerability dataset, which includes C / C++ function source code, vulnerability tags, vulnerable line annotations, vulnerability descriptions, general vulnerability enumeration categories, and publicly available vulnerability numbers.
[0051] The core objective of data preprocessing is to select high-quality function-level data samples suitable for subsequent inference and interpretation generation and model training. To ensure that the training samples possess complete vulnerability analysis semantics, this invention applies three types of preprocessing rules to the original function-level samples:
[0052] Rule 1: Code Semantic Filtering. The system removes function samples that are weakly related to vulnerability analysis, such as functions that only contain constant definitions, parameter configurations, log outputs, debugging information, user interface construction logic, or simple encapsulated calls. Simultaneously, the system filters vulnerability samples that lack vulnerability categories or publicly disclosed vulnerability identifiers to reduce annotation noise caused by missing security semantics.
[0053] Rule 2: Function Length Constraint. Based on the context window and training resource limitations of the target large language model, the system performs a length check on the function source code, retaining only function samples that do not exceed a preset length. In one embodiment, the preset length is 2048 tags; functions exceeding this length will be discarded to ensure all samples can be fully input into the model. This rule ensures that the function length does not exceed the context window limit of the target large language model, avoiding training truncation, context loss, or excessive memory overhead caused by overly long functions.
[0054] Rule 3: Semantic Integrity Check. The system determines whether a function sample can independently express the complete functionality of the program. Samples that heavily rely on external context, have incomplete syntax, missing function bodies, or cannot support basic vulnerability semantic analysis are filtered out. After this step, the remaining samples can provide a relatively complete code context for subsequent reasoning and interpretation.
[0055] Through the above preprocessing, the system obtains function-level code samples that retain complete vulnerability semantics, providing a foundation for the subsequent construction of a reasoning and interpretation dataset.
[0056] (2) Reasoning and Explanation Text Generation and Verification
[0057] To construct a high-quality training dataset with stepwise inference explanations suitable for supervised and reinforcement fine-tuning of large language models, and to address the scarcity of high-quality vulnerability inference data, this embodiment designs a two-stage teacher model generation and validation framework. This framework includes an initial inference explanation generation stage and an inference explanation quality validation stage.
[0058] The first stage is the generation of initial inference explanations. The system inputs the source code, vulnerability labels, and structured hints (Prompt1) into the first teacher model, requiring the teacher model to output a fixed-formatted step-by-step inference process (Step 1, Step 2,...), a final conclusion (the code is safe or the code is vulnerable), and a structured result. The structured result includes function information, vulnerability category, vulnerability description, detection target, vulnerable lines, and inference explanation text. For samples with vulnerabilities, the structured hints further include vulnerability category and vulnerability description, ensuring that the generated inference explanation revolves around vulnerability triggering patterns, key variables, dangerous calls, boundary conditions, resource management, or control flow conditions. To reduce formatting errors and illusions in the content generated by the teacher model, this embodiment introduces an iterative self-checking mechanism in the first stage. The system uses a predefined structure template to check whether the generated content contains necessary fields and whether the inference steps are consistent with the final vulnerability label. If the generated result does not meet the structure template or the conclusion is inconsistent with the label, regeneration is triggered (combined with Prompt2 for few-sample structure correction, or combined with Prompt3 for enhanced correction using vulnerability description and category). This mechanism ensures that the generated text maintains a consistent format and reduces erroneous interpretations that conflict with the original vulnerability labels. After self-checking, Prompt 4 is used to allow the model to perform a final verification of the inference explanation text, ensuring correct formatting, logical coherence, and consistency between conclusions and labels. This stage ultimately outputs the preliminarily approved inference explanation text.
[0059] The second stage is the verification of the reasoning explanation quality. The system inputs the reasoning explanation text generated in the first stage and which has passed the self-check into the second teacher model, which performs a double consistency audit, including semantic consistency audit and security semantic consistency audit.
[0060] Semantic consistency audit (using Prompt 5) checks whether each inference step corresponds to a specific program statement, whether it conforms to the actual code semantics of the function, whether there are fictitious variables, fictitious calls, or generalized descriptions that deviate from the code, and outputs Semantic consistency: valid / invalid and the reason for the inconsistency.
[0061] Security semantic consistency audit (using Prompt 6) checks whether the reasoning explanation meets the requirements of the vulnerability detection task: For vulnerability samples, the system checks whether the reasoning explanation, combined with the vulnerability category, vulnerability description, and code triggering mode, locates a reasonable risk; for non-vulnerable samples, the system checks whether the reasoning explanation can explain why the current code does not exhibit the corresponding security flaw. During the verification process, the system keeps the original vulnerability labels unchanged to avoid label drift caused by the teacher model changing the labels itself. The final output is "Security matching: valid / invalid" and the verification reason.
[0062] Using the two-stage framework described above, the system makes a final verification decision (using Prompt 7) to determine whether the reasoning explanation is acceptable, and outputs the final decision: VALID / INVALID and the reason.
[0063] This step uses double-verified reasoning to interpret the text, along with the original source code and vulnerability labels, to form a reinforced fine-tuning dataset.
[0064] (3) Supervise and fine-tune the preheating
[0065] Directly training a pre-trained large language model using reinforcement learning can easily lead to training instability, an excessively large exploration space, and convergence difficulties. Therefore, this embodiment first performs supervised fine-tuning on a reasoning explanation dataset to enable the model to possess basic vulnerability detection capabilities and structured reasoning explanation generation capabilities.
[0066] Using the high-quality dataset produced in the previous stage (inference explanation text generation and verification), each data point contains: source code C, inference explanation text T, and vulnerability label L (binary label, 1 indicates a vulnerability, 0 indicates security). T and L are concatenated into a complete target output sequence Y=[T;L]. The target output sequence includes the progressive inference explanation and the final binary vulnerability label that the model needs to generate.
[0067] The training method employs standard autoregressive language model training. The input source code C (formatted and packaged, such as in the style of Prompt 1 or Prompt 10) outputs the target sequence Y. The loss function is cross-entropy loss, which maximizes the probability of generating the target sequence given the source code, i.e., minimizing the cross-entropy loss. Specifically, at each decoding step, the model predicts the next token based on the input source code and previously generated tokens (a sequence of all tokens output by the model before the current decoding step). The model's internal Transformer neural network calculates a probability score for all possible candidate tokens. This score represents "how likely the next token is a certain word in the current context." Based on the calculated probability distribution, the model selects a token as the output. The newly selected token is appended to the end of the historical token sequence, and the process moves to the next step to predict the next token. This process is repeated until the model outputs a specific end token or reaches the maximum length.
[0068] After supervised fine-tuning is completed, the system obtains the initial policy model. This initial policy model serves as the starting point for subsequent reinforcement learning stages and also as a reference policy, used to calculate the deviation between the current policy and the initial reference policy during the reinforcement fine-tuning stage.
[0069] (4) Reinforcement learning online sampling
[0070] During the reinforcement learning phase, the system performs online sampling based on the initial policy model obtained through supervised fine-tuning. First, for each input code sample C, the current policy model is... Set to build mode, for each source code C i The complete output sequence Y is generated autoregressively. i The sequence includes progressively reasoned explanation text and final vulnerability prediction labels. Due to the sampling randomness introduced during the decoding phase, the same source code C... i Inputting the same data and generating it multiple times will result in slightly different output sequences Y. i Typically, G candidate outputs, called a sampling group, are generated for each source code. Then, a sequence Y is generated for each output in the sampling group. i Calculate a scalar reward value using the previously designed composite reward function.
[0071] The three reward categories constrain the model output from three levels: the correctness of the detection results, the consistency of the inference path, and the semantic consistency of the overall interpretation. Next, the model parameters are updated. The obtained (source code, generated sequence, reward) is used as the training signal, and the policy gradient is calculated using a reinforcement learning algorithm to update the model parameters θ. After the parameter update, the model's behavior distribution has changed. The next iteration must use the new π. θ Re-perform sampling.
[0072] (5) Results monitoring and rewards
[0073] The resulting supervisory rewards are used to directly supervise the model's binary classification of vulnerabilities. Through an asymmetric reward and penalty design, the model is guided to learn detection strategies that align with the risk preferences of the security scenario. The system compares the predicted labels generated by the model with the actual labels and assigns different rewards based on four scenarios: a first positive reward is given when both the prediction and the actual label is a vulnerability; a second positive reward is given when both the prediction and the actual label is not a vulnerability; a first negative reward is given when both the prediction and the actual label is not a vulnerability; and a second negative reward is given when both the prediction and the actual label is not a vulnerability. The first positive reward is greater than the second positive reward, and the penalty for the second negative reward is greater than that for the first negative reward. This design reflects the asymmetric risk-cost relationship in software security scenarios: correctly identifying vulnerabilities is more important than correctly identifying benign code, and failing to report a real vulnerability is more dangerous than falsely reporting benign code.
[0074] As shown in the following formula, in one embodiment, the result supervision reward function is expressed as follows: The first positive reward is +1.0, the second positive reward is +0.5, the first negative reward is -0.2, and the second negative reward is -1.0. This reward function can alleviate the problems of reward sparsity and category risk asymmetry in vulnerability detection, guiding the model to pay more attention to real vulnerability samples.
[0075]
[0076] (6) Process structure perception reward
[0077] When generating vulnerability reasoning explanations, language models are prone to deviating from the actual semantics of the source code, leading to problems such as illusions, inconsistent paths, or incomplete logic. For example... Figure 3 As shown, the process structure-aware reward is used to supervise whether the model pays attention to the real code structure and vulnerability dependency paths in each step of the reasoning explanation generated. By transforming the real structural information of the code (variables, functions, dependencies) into quantifiable reward signals, the model is guided to pay attention to the truly important keywords and dependency paths in the code when generating each reasoning step.
[0078] For a vulnerable function, the system first constructs a code attribute graph for the function and extracts program dependencies from the graph. Then, starting from the vulnerable behavior, the system performs a backward program slice to obtain the vulnerable statement, first-level dependency statements, and second-level dependency statements. The variable names and function names in the vulnerable statement form the zero-level keyword set K0, the variable names and function names in the first-level dependency statements form the first-level keyword set K1, and the variable names and function names in the second-level dependency statements form the second-level keyword set K2. To differentiate the importance of different levels, the system assigns different weights to the three keyword levels. and satisfy Statements closer to the vulnerability trigger point are given greater weight. The unit reward value for the k-th layer keyword... Defined as:
[0079]
[0080] in Indicates the total reward scale. and These represent the number of keyword types and the total number of occurrences at the k-th level, respectively.
[0081] This design prioritizes the vulnerability statement itself, followed by direct dependencies, and then indirect dependencies. For each inference step generated by the model, the system extracts variable names, function names, and code-related keywords to form a step keyword set. Subsequently, the system calculates the frequency overlap between the step keyword set and K0, K1, and K2, and obtains the step reward according to the layer weights. The step rewards for all inference steps are averaged to obtain the process structure-aware reward. Let the generated inference content contain N inference steps, and let the keyword set extracted in the nth step be denoted as... The reward for this step of reasoning and the total reward for the entire reasoning process can be represented as follows:
[0082]
[0083]
[0084] in Keywords The number of times it appears in set X This indicates that all keywords are traversed. Used to count the number of valid matches between the current inference step and the corresponding dependency layer.
[0085] For non-vulnerable functions, since no vulnerable lines exist, the system extracts representative variable names and function names based on the program structure information in the code attribute graph, forming a set of non-vulnerable structure keywords. Then, the system calculates the keyword frequency overlap between the inference steps generated by the model and the set of non-vulnerable structure keywords, obtaining a process structure-aware reward for the non-vulnerable sample. This method makes the process structure-aware reward applicable not only to vulnerable samples but also to the quality evaluation of security analysis and interpretation of benign samples.
[0086] For keyword extraction from both vulnerability and non-vulnerability samples, please refer to Prompt 8, the K1 / K2 Dependency Keyword Extraction Process Description, and Prompt 9, respectively.
[0087] (7) Overall semantic consistency reward
[0088] The overall semantic consistency reward is used to constrain the global semantic consistency between the inference explanation text generated by the model and the reference inference explanation text, and to prevent the model from obtaining process structure rewards by simply piling up key variables and function names.
[0089] The system designates the inference explanation text generated by the enhanced fine-tuning model as the generated explanation and the inference explanation text generated by the teacher model as the reference explanation. The system uses a sentence embedding model to obtain the semantic representations of the generated and reference explanations respectively. In this embodiment, the sentence embedding model uses all-mpnet-base-v2.
[0090] Because the reasoning explanation text can be quite long, the system employs a sliding window semantic aggregation strategy, dividing the generated explanation and the reference explanation into multiple aligned text segments. Subsequently, the system calculates the cosine similarity between the semantic vectors of each group of aligned segments and takes the average as the overall semantic consistency reward. and Divide into N aligned subsequences based on a fixed window size. and This addresses the issue of input length limitations in encoding models. The overall semantic reward is defined as the average cosine similarity of each window:
[0091]
[0092] Right now:
[0093]
[0094] Where E(⋅) represents the sentence encoding function.
[0095] This reward can constrain the semantic direction of reasoning and interpretation at the overall level. When the text generated by the model contains key variables or functions, but the overall analysis deviates from the reference interpretation or lacks vulnerability semantics, the overall semantic consistency reward will decrease, thereby suppressing reward theft.
[0096] (8) Composite rewards and reinforcement fine-tuning
[0097] like Figure 2 As shown, the system normalizes the result supervision reward, process structure awareness reward, and overall semantic consistency reward, and then weights and fuses them according to preset weights. Subsequently, the system subtracts the divergence penalty between the current policy and the initial reference policy to obtain the composite reward. The composite reward consists of four parts: result supervision reward... Process structure perception reward Overall semantic consistency reward and strategy deviation penalty In this embodiment, the composite reward can be represented as:
[0098]
[0099] in, To reward the balance coefficient. β represents the currently learned policy; β is the KL divergence penalty coefficient. The KL divergence term is used to limit the current policy from deviating too quickly from the initial policy, thereby improving the stability and generalization ability of the reinforcement learning training process.
[0100] In this embodiment, the system employs the Proximal Policy Optimization (PPO) algorithm to update model parameters and calculates advantage estimation based on compound rewards. PPO is a widely used reinforcement learning algorithm with advantages such as stable training and ease of implementation. Its core idea is to prevent performance collapse caused by excessively large single updates by clipping the magnitude of policy updates. To reduce training resource requirements, the system uses a low-rank adaptation method for efficient parameter fine-tuning, updating only a portion of the adapted parameters instead of all model parameters. The system can also attach a linear value head to the last hidden state of the policy model to estimate the value function and calculate the value loss. The low-rank adaptation rank, learning rate, batch size, reward weight, and KL penalty coefficient can be adjusted according to model size, training data, and computational resources.
[0101] (9) Vulnerability detection and inference output
[0102] After fine-tuning, the system obtains a target model for vulnerability detection and inference. During the inference phase, the system converts the source code to be detected into structured prompts and inputs them into the target model. The target model performs a single forward inference, outputting the inference explanation text, vulnerability description, and binary vulnerability prediction labels. This inference phase only requires inputting inference prompts and the source code to be detected; it does not require inputting actual vulnerability labels, vulnerability descriptions, or vulnerability categories. See Prompt 10 for specific inference prompts.
[0103] The reasoning explanation text describes key variables, key functions, potentially risky statements, dependency paths, vulnerability triggering conditions, and the final judgment reasons in the code; the vulnerability description is used to help security analysts understand the vulnerability type and impact; and the binary vulnerability prediction label is used to indicate whether the code to be detected has a vulnerability.
[0104] The prompt word template Prompt1-10 used in this embodiment:
[0105] Prompt 1: Initial inference explanation to generate a Prompt template You are a C / C++ code security analysis expert. Your task is to perform a step-by-step security analysis of a given function and its vulnerability labels, and generate inference and explanation text. A vulnerability label of 1 indicates that the function contains a vulnerability, and a vulnerability label of 0 indicates that the function is secure.
Input Content
Output Requirements
Notes
Input Content
Output Requirements
Input Content
Notes
Input Content
Output Requirements
Input
Output Format
Input Content
Output Format
Input Content
Output Format
Input
Output Format
Notes
Processing Flow
Input Content
Output Format
Notes
Input Content
[0106] Terminology Explanation
[0107] the term meaning Large Language Model It refers to a large-scale neural network model that can process natural language and program code, and can generate code analysis, reasoning interpretation and prediction results based on input prompts. Supervision and fine-tuning This refers to a training method that uses labeled data with target output to train a base model, enabling the model to learn the mapping relationship between a given input and a standard output. Strengthen fine-tuning This refers to a training method that uses a reward function to score the results generated by the model and updates the model parameters through reinforcement learning algorithms to make the model output more in line with the specific task objectives. Inference and Explanation Text This refers to the step-by-step analysis process generated by the model when determining whether there are vulnerabilities in the code. It is used to explain the cause of the vulnerability, the triggering conditions, the key statements, the dependency paths, and the final judgment reasons. Code Attribute Diagram It refers to a graph structure that uniformly represents the abstract syntax structure of code, control flow relationships, and data dependencies. Process structure perception reward This refers to a reward mechanism that scores the reasoning steps generated by the model based on the key variables and functions in the vulnerability statement and its dependent statements.
[0108] Example 2
[0109] This embodiment follows the sequence of "constructing the VDCoT reinforcement fine-tuning dataset from the BigVul source dataset, supervised fine-tuning warm-up, reinforcement fine-tuning training, detection inference, experimental effect verification, and case study".
[0110] (1) Constructing a reinforcement fine-tuning dataset with inference explanation text based on bigvul
[0111] In one embodiment, the system uses the publicly available function-level vulnerability dataset BigVul as the source dataset. This dataset contains C / C++ function-level source code samples from multiple open-source projects and provides fields such as vulnerability tag, vulnerability line, patch submission, vulnerability description, general vulnerability enumeration category, and publicly available vulnerability number. The system first performs code semantic filtering, function length constraints, and semantic integrity checks on the original BigVul samples to obtain function-level code samples that retain complete vulnerability semantics.
[0112] After preprocessing, the system employs a two-stage teacher model generation and verification framework to generate inference explanation text. In the first stage, the system inputs the function source code, vulnerability label, vulnerability description, and vulnerability category into the first teacher model, which generates structured inference explanation text. In the second stage, the system inputs the inference explanation text generated in the first stage and passing self-checks into the second teacher model, which performs semantic consistency verification and security semantic consistency verification. In this stage, the system keeps the original vulnerability labels unchanged to prevent label drift caused by the teacher model changing the labels on its own.
[0113] Through the above process, the system obtains the reinforcement fine-tuning dataset VDCoT with inference explanation text. In one embodiment, VDCoT contains 19,149 function-level C / C++ code samples, of which 9,569 are vulnerability samples and 9,580 are non-vulnerable samples, and is divided into training set, validation set and test set in an 8:1:1 ratio.
[0114] (2) Strengthen fine-tuning
[0115] After obtaining the VDCoT dataset, the system first performs supervised fine-tuning on the basic large language model for warm-up. The system takes the source code as input and combines the inference explanation text and the final vulnerability label combination as the target output sequence. This allows the model to learn to generate a structured vulnerability analysis process based on the function source code and output binary vulnerability labels in the conclusion. After supervised fine-tuning, the system obtains an initial policy model. This initial policy model can generate basically usable inference explanation text and serves as the initial and reference policy for the reinforcement learning phase.
[0116] During the fine-tuning phase, the system performs online sampling based on the initial policy model. For each input code sample, the current policy model generates a complete output sequence, which includes progressive inference explanation text and final vulnerability prediction labels. The system parses the prediction labels and inference explanation text from the output sequence and calculates the result supervision reward, process structure awareness reward, and overall semantic consistency reward, respectively. The system normalizes and weights these three types of rewards and subtracts the policy deviation penalty of the current policy relative to the initial reference policy to obtain the composite reward. Subsequently, the system calculates the advantage estimate and updates the model parameters based on the near-end policy optimization algorithm.
[0117] (3) Detection and reasoning deployment
[0118] After the enhancement and fine-tuning are completed, the system obtains an enhanced and fine-tuned model for vulnerability detection and inference interpretation. During the inference phase, the user inputs the source code of the function to be detected. The system fills the source code into a structured prompt template and inputs it into the enhanced and fine-tuned model. The model performs a single forward inference, outputting progressive inference interpretation text, vulnerability description, and binary vulnerability prediction labels. This inference phase does not require inputting vulnerability labels, vulnerability descriptions, or vulnerability categories, nor does it require additional calls to the teacher model; the inference phase only requires inputting the detection inference prompt words shown in Prompt 10 and the code to be tested.
[0119] (4) Comparison of detection results with baseline methods
[0120] To verify the detection performance of the method of the present invention, in one embodiment, the method of the present invention is compared with multiple vulnerability detection baseline methods on three datasets: VDCoT, Devign, and Reveal. The comparison objects include traditional deep learning or graph neural network methods, vulnerability detection methods based on large language models, and general closed-source large language model methods. Table 1 below lists the representative baseline results most relevant to the present invention.
[0121] Table 1. Comparison of detection performance with representative baseline methods (unit: %)
[0122] method VDCoT F1 score DevignF1 score Reveal F1 score Average accuracy Average recall Average F1 score VulLLM 88.11 50.68 30.70 65.77 72.04 56.50 M2CVD 72.44 57.88 38.87 52.84 64.99 56.55 RDVulD (SFT only) 75.87 52.77 39.28 62.60 53.88 55.97 Method of the present invention 95.47 64.74 53.39 79.59 66.70 71.20
[0123] As shown in Table 1, the average Precision, Recall, and F1-score of the method of this invention on the three datasets are 79.59%, 66.70%, and 71.20%, respectively. Compared with the state-of-the-art baseline method M2CVD with an average F1-score of 56.55%, the method of this invention improves the average F1-score by 14.65 percentage points, a relative improvement of 25.91%; compared with VulLLM with an average F1-score of 56.50%, the method of this invention improves the average F1-score by 14.70 percentage points, a relative improvement of 26.02%. This indicates that the method of this invention not only improves the performance of function-level vulnerability detection but also has good generalization ability on different datasets.
[0124] (5) Ablation experiment of reward module
[0125] To verify the roles of different reward modules in this invention, in one embodiment, ablation analysis was performed on the process structure awareness reward, the overall semantic consistency reward, and the result supervision reward. The detection performance ablation results are shown in the table below.
[0126] Table 2. Vulnerability detection results after ablation of the reward module (unit: %)
[0127] Model settings Average accuracy Average recall Average F1 score Remove process structure-aware reward (w / o PSAR) 80.92 49.13 57.92 Remove the overall semantic consistency reward (w / o SCR) 73.80 62.26 67.99 Simultaneously remove PSAR and SCR 77.88 49.48 57.32 Replace with ordinary mathematical reasoning reward 80.64 43.77 52.97 Method of the present invention 79.59 66.70 71.20
[0128] As shown in Table 2, removing the process structure awareness reward reduces the average F1-score to 57.92%; removing both the process structure awareness reward and the overall semantic consistency reward reduces the average F1-score to 57.32%; and replacing the asymmetric result supervision reward in this invention with a regular mathematical reasoning reward reduces the average F1-score to 52.97%. These results demonstrate that the multi-dimensional reward mechanism in this invention is not simply additive, but rather constrains the model output from three levels: classification results, structural path, and overall semantics, thereby collectively improving vulnerability detection performance.
[0129] (6) Case study of CVE-2015-4003
[0130] To qualitatively demonstrate the advantages of the present invention's method over supervised baseline fine-tuning in understanding code semantics and vulnerability root causes, this embodiment selects the vulnerable function oz_cdev_write corresponding to CVE-2015-4003 for a case study, such as... Figure 4 As shown. This function handles character device write operations, where the user-controllable `count` parameter is involved in the length field calculation, which poses a risk of arithmetic overflow.
[0131] In this code, the critical vulnerable statement is line 31: "elt->length = sizeof(struct oz_app_hdr) + count;". When the user-controlled value of count is too large, the result of this addition may cause an integer overflow, which in turn affects subsequent length field and buffer operations, leading to kernel memory corruption or denial-of-service risks.
[0132] To preserve the readability of the model's generated results, the key reasoning content of the two models is presented below in excerpt form. As shown in Tables 3 and 4, while the SFT model can correctly output vulnerability labels, it incorrectly attributes the root cause of vulnerabilities to race conditions and generates explanations inconsistent with the facts in the source code. For example, the SFT model claims that the function did not verify whether the assignment was successful, even though the source code already contains an "if (ei == NULL)" check. This phenomenon indicates that when relying solely on supervised fine-tuning, the model may learn superficial statistical patterns without truly reasoning based on control flow and data dependencies.
[0133] In contrast, the method of this invention can locate the critical vulnerable statement "elt->length = sizeof(struct oz_app_hdr) + count", and analyzes it around the user-controllable parameter count, the calculation of the length field, and the subsequent user-mode copy operation. Its reasoning process strictly follows the actual execution order of the function, unfolding step by step from lock acquisition, state checking, memory allocation, length calculation, and user copying to the queue operation. This demonstrates that process structure-aware rewards can enhance the model's attention to code dependency paths, and overall semantic consistency rewards can reduce root cause misjudgments and interpretive illusions.
[0134] Table 3 Comparison of inference interpretation between the supervised fine-tuning model SFT and the method of this invention in the CVE-2015-4003 case.
[0135] Comparison items Supervised fine-tuning model (SFT) The method of this invention (RDVulD) Vulnerability Tags It was identified as a vulnerability, but the final label was correct. It was identified as a vulnerability, but the final label was correct. Root cause localization The root cause error was attributed to CWE-362, which is improper synchronization or race conditions in concurrent execution of shared resources. The length calculation statement elt->length=sizeof(struct oz_app_hdr) + count was located and identified as an integer overflow risk. Reasoning path There are illusions and root cause misjudgments, such as claiming that the function did not verify whether the allocation was successful, but the source code contains an ei == NULL check. The analysis is conducted according to the actual execution order of lock acquisition, state verification, memory allocation, length calculation, user copying, and queue operations. Explain quality Although the prediction was correct, the reasoning and explanation were not rooted in the actual code path, making it difficult to support root cause auditing of vulnerabilities. The explanation can be traced back to the specific vulnerable statement and the variable count, which can illustrate the potential triggering conditions and impact.
[0136] Table 4. Excerpts of the reasoning explanation text generated by the two models
[0137] SFT model inference results excerpt Excerpt of the reasoning results of the method of this invention Final label: Vulnerability (label correct). Key reasoning: Step 4 states "the function did not verify successful allocation," but the source code clearly shows an `ei == NULL` check. Step 5 describes the problem as `copy_from_user` not verifying the user buffer size. Step 8 attributes the root cause of the vulnerability to a CWE-362 race condition. Conclusion: The label is correct, but the root cause is misjudged and there is an interpretation inconsistent with the code control flow. Final Label: Vulnerability (Label Correct). Key Reasoning: Step 5: Accurately locate elt->length = sizeof(struct oz_app_hdr) + count. Step 6: Further explain that an excessively large count may affect memory access related to copy_from_user. Step 12: Infer that an attacker can trigger a kernel crash or denial-of-service attack through a large count value. Conclusion: Label correct, vulnerability statement accurately located, reasoning path consistent with function execution order.
[0138] Therefore, this case demonstrates that the method of the present invention can not only provide correct binary vulnerability labels, but also generate reasoning and explanation text that is more consistent with the actual code path and helps security personnel understand the root cause of the vulnerability.
[0139] It should be noted that, to avoid limiting the scope of protection of this invention to a specific experimental configuration, the teacher model, base model, sentence embedding model, low-rank adaptation parameters, reinforcement learning algorithm parameters, and data sources in this invention can all be replaced according to actual engineering conditions. The first teacher model and the second teacher model can be large language models with code understanding and natural language reasoning capabilities; the sentence embedding model can be a model capable of outputting text semantic vectors; the function-level vulnerability data can come from public datasets, or from internal code audit samples or manually annotated samples.
[0140] Without altering the core scheme of "inference interpretation data construction, supervised fine-tuning warm-up, multi-dimensional reward-driven reinforcement fine-tuning, and joint output of vulnerability detection and inference interpretation," any substitution or equivalent transformation of specific models, parameters, and tools falls within the feasible scope of this invention's technical concept.
Claims
1. A software vulnerability detection and reasoning explanation method based on a reward-driven large language model, characterized in that, Includes the following steps: S1, acquire function-level vulnerability sample data, and perform code semantic filtering, function length constraint and semantic integrity check on the sample data to obtain preprocessed sample data; S2, using a two-stage teacher model generation and verification framework, automatically generates a training dataset containing stepwise reasoning explanation text and vulnerability labels for the preprocessed sample data; S3, Use the training dataset to perform supervised fine-tuning of the basic large language model, so that the model learns the mapping relationship from source code to inference explanation text and vulnerability labels, and obtains the initial policy model; S4, starting with the initial policy model, an output sequence is generated through online sampling, and the model is trained using reinforcement learning based on a composite reward function to obtain a reinforcement fine-tuning model; wherein, the composite reward function includes at least a result supervision reward for evaluating the correctness of the vulnerability binary classification result. The process structure-aware reward used to evaluate the consistency between the reasoning explanation text and the actual dependency path of the code in the output sequence. And a global semantic consistency reward used to evaluate the global semantic consistency between the inference explanation text and the reference inference explanation text in the output sequence. ; S5, input the source code to be detected into the enhanced fine-tuning model, and the model outputs the detection result containing the stepwise reasoning explanation text, vulnerability description and binary vulnerability prediction label in a single forward inference.
2. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 1, characterized in that, Step S2 includes: inputting source code, vulnerability labels, and structured hints into the first teacher model to generate a structured result containing the step-by-step reasoning process, and performing format checks and conclusion consistency self-checks on the generated result, triggering regeneration for results that do not meet the requirements; inputting the reasoning explanation text that passes the self-check into the second teacher model to perform semantic consistency audit and security semantic matching audit; wherein, the semantic consistency audit is used to check whether the reasoning steps correspond to specific code statements and whether there are fictitious variables or functions, and the security semantic matching audit is used to check whether the reasoning explanation conforms to the vulnerability category and vulnerability description and does not change the original vulnerability label; the reasoning explanation text that passes the dual audit, together with the corresponding source code and vulnerability labels, constitutes the training dataset.
3. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 1, characterized in that, In step S4, the result is supervised and rewarded. The calculation method is as follows: the vulnerability labels predicted by the model are compared with the actual labels. Based on four scenarios, namely, the predicted vulnerability is a vulnerability and it is a vulnerability, the predicted non-vulnerability is a non-vulnerability, the predicted vulnerability is a vulnerability and it is a non-vulnerability, and the predicted non-vulnerability is a vulnerability and it is a vulnerability, a first positive reward, a second positive reward, a first negative reward, and a second negative reward are assigned respectively. The first positive reward is greater than the second positive reward, and the penalty intensity of the second negative reward is greater than the penalty intensity of the first negative reward.
4. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 1, characterized in that, In step S4, the process structure perception reward The calculation method is as follows: For a vulnerability sample, a code attribute graph is constructed for the function, and a backward program slice is executed starting from the vulnerability behavior. Variable names and function names in the vulnerability statement are extracted to form a zero-level keyword set. Variable names and function names in the first-level dependency statement are extracted to form a first-level keyword set. Variable names and function names in the second-level dependency statement are extracted to form a second-level keyword set, and decreasing layer weights are assigned to each level. Variable names, function names, and code-related keywords are extracted from each inference step generated by the model to form a step keyword set. The frequency overlap between the set of keywords for each step and the set of keywords for each level is calculated, and the result is weighted and summed according to the level weights to obtain the step reward for that reasoning step; the average of the step rewards for all reasoning steps is taken to obtain the process structure perception reward.
5. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 4, characterized in that, For non-vulnerable samples, representative variable names and function names are extracted based on the program structure information in the code attribute graph to form a non-vulnerable structure keyword set. The inference steps generated by the computation model overlap with the keyword frequency of the non-vulnerable structure keyword set to obtain the process structure perception reward of the non-vulnerable sample.
6. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 4 or 5, characterized in that, The process structure perception reward Represented as: , , , The generated reasoning content contains N reasoning steps. The reward for reasoning in step n. Keywords The number of times it appears in set X This indicates that all keywords are traversed. Used to count the number of valid matches between the current inference step and the corresponding dependency layer; The unit reward value for the keyword at the k-th level. Indicates the total reward scale. and These represent the number of keyword types and the total number of occurrences at the k-th level, respectively.
7. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 1, characterized in that, In step S4, the overall semantic consistency reward The calculation method is as follows: the reasoning explanation text generated by the model is recorded as the generated explanation, and the reasoning explanation text generated by the teacher model is recorded as the reference explanation; The semantic vector representations of the generated explanation and the reference explanation are obtained by using a sentence embedding model. The generated explanation and the reference explanation are divided into multiple aligned text segments by using a sliding window semantic aggregation strategy. The cosine similarity between the semantic vectors of each group of aligned segments is calculated. The average cosine similarity of all aligned segments is used to obtain the overall semantic consistency reward.
8. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 7, characterized in that, The sentence embedding model is all-mpnet-base-v2; and / or the overall semantic consistency reward. Represented as: Where E(⋅) represents the sentence encoding function, and Let i represent the i-th predicted text and the i-th reference text, respectively.
9. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to any one of claims 1-8, characterized in that, The composite reward function is expressed as follows: , in, To reward the balance coefficient, The strategy learned so far, Let β be the initial reference strategy and β be the KL divergence penalty coefficient.
10. The software vulnerability detection and reasoning explanation method based on a reward-driven large language model according to claim 1, characterized in that, In step S4, the reasoning and interpretation text analyzes key variables, key functions, input boundaries, memory operations, resource management, control flow conditions, and data dependencies step by step according to the actual execution order of the functions. If a vulnerability is found, the vulnerable statement, vulnerability type, triggering condition, and impact are pointed out. If no vulnerability is found, the key security checks, boundary constraints, error handling, or resource management logic are explained.
