A reinforcement learning training method for improving vulnerability repair model code structure perception ability

Abstract

The application discloses a reinforcement learning training method for improving the code structure perception ability of a vulnerability repair model, belongs to the field of software security and deep reinforcement learning, and comprises the following steps: step 1, a multi-angle structure similarity reward function is designed; step 2, a Critic model is designed; and step 3, a reinforcement learning process is designed. Through the above method, the code structure perception ability of the repair model is enhanced, so that a patch closer to the real label is generated.

Description

Technical Field

[0001] This invention relates to a reinforcement learning training method for improving the code structure awareness capability of vulnerability repair models, belonging to the fields of software security and deep reinforcement learning. Background Technology

[0002] Traditional methods of handling software vulnerabilities manually face numerous challenges: on the one hand, manual vulnerability patching is time-consuming and error-prone; on the other hand, manual intervention is not only difficult during the patching process but may also unintentionally introduce new vulnerabilities. With the development of deep learning and the increase in open-source models, researchers have begun to apply deep learning methods to automated code vulnerability patching research. The most typical approach is to treat code as natural language, viewing the vulnerability patching task as a sequence-to-sequence task. That is, the code containing the vulnerability is input into a "translation" model, which outputs another set of patch sequences. Common Neural Machine Translation (NMT) models typically parse the source code into a sequence of tokens and apply a word vector model to each token, mapping this sequence to a vector space. Then, the encoder learns a deep representation, and the decoder decodes it, outputting the corresponding vulnerability patch. Although deep learning models have made progress in understanding the relationships between tokens in code, the application of this method in the field of automated program patching still faces some challenges. The main problem is that while these models can capture certain relationships between code tokens, they often lack a deep understanding of the structural information such as code syntax and semantics. This means that although models may identify certain patterns in the code, the patches they generate may not fully conform to the syntax rules of the programming language, or may not accurately fix the actual problems in the code.

[0003] RLHF (Reinforcement Learning from Human Feedback) is a fine-tuning method combining supervised learning and reinforcement learning. It aims to optimize model output using human preference data to better align with human values ​​and practical needs. Its core process consists of three steps: First, candidate answers are generated based on a pre-trained model (such as the GPT series), and human feedback on answer quality is collected through manual annotation or comparative ranking. Then, this feedback is used to train a reward model, transforming subjective human preferences into a quantifiable scoring function. Finally, the original model is fine-tuned using reinforcement learning algorithms (such as PPO) to maximize the reward model's score, while constraints such as KL divergence prevent the model from deviating excessively from its initial state. RLHF has shown outstanding performance in dialogue systems and content generation scenarios; for example, ChatGPT significantly improved the safety, relevance, and logical consistency of responses using this method. Its advantages lie in its ability to handle complex and fuzzy optimization objectives (such as "natural conversational feel") and reduce reliance on large amounts of manually labeled data. However, it also faces challenges: human feedback may be subjective and inconsistent, potentially introducing implicit biases; the accuracy of the reward model directly affects fine-tuning, and over-optimization may lead to rigid model output or deviate from its original intent. Furthermore, RLHF requires multi-stage training and substantial computational resources, placing high demands on algorithm design and engineering implementation. Nevertheless, it remains one of the core technologies for aligning AI behavior with human intent. Summary of the Invention

[0004] The purpose of this invention is to provide a reinforcement learning training method to improve the code structure awareness capability of vulnerability remediation models. This method solves the technical problem of insufficient code structure awareness capability of existing vulnerability remediation models and improves the structural similarity between patches generated by the remediation model and real labels.

[0005] The technical solution adopted in this invention is as follows: A reinforcement learning training method to improve the code structure awareness capability of a vulnerability repair model:

[0006] Step 1: Design a multi-angle structural similarity reward function:

[0007] 1.1) Calculate the positive reward function R + ;

[0008] Positive reward function R + The calculation method is as follows:

[0009] 1.1.1) CodeBLEU

[0010] CodeBLEU=α·BLEU+β·BLEU weight +γ·Match ast +δ·Matchdf (1)

[0011] Where α, β, γ, and δ are weighting coefficients, and BLEU represents the standard BLEU and BLEU. weight For weighted BLEU, Match ast Matching percentage of AST subtrees df The proportion of data stream matching;

[0012] 1.1.2) T5Score

[0013] 1.1.2.1) Token representation: Assuming the synthesized patch code after generation is represented as x = {x1, x2, ..., x} k}, the reference code is Using the SASTCodeT5 encoder, x and... Generate context embeddings x = {x1, x2, ..., x} k}and

[0014] 1.1.2.2) Calculate similarity: Calculate the similarity between the context embeddings of the vector representations, x i and The cosine similarity is By normalizing, the calculation is simplified to an inner product.

[0015] 1.1.2.3) Calculate precision and recall: The full T5Score uses greedy matching to match each token in x with... Token matching in the middle is used to calculate recall, and Each token in the string is matched against one of the tokens in the string to calculate precision, and each token is matched against the most similar token in another string; then BLEU in CodeBLEU is used. weight The weighting method w(x) i This means setting the keyword weight to 2.5 times that of other keywords, with precision and recall rates as shown in the formulas:

[0016]

[0017] Among them, R T5 P represents recall rate. T5 Indicate precision;

[0018] 1.1.2.4) Scaling: To scale the cosine similarity from -1 to 1 to between 0 and 1, calculate R from the previous step. T5 P T5 The average value, let R be. T5 The average value is a, PT5 The average value is b, which, after scaling, is:

[0019]

[0020] 1.1.2.5) Calculate T5Score

[0021]

[0022] 1.1.3) Tree edit distance TED

[0023] Dynamic programming is used to calculate the minimum number of node deletions, insertions, and replacements required to move from one tree to another, thus introducing TED (Thinking-Based Tree) into the evaluation of Abstract Syntax Trees (ASTs).

[0024] In TED, trees and forests, edit operations and edit scripts, and tree edit distances are defined. For a tree on the alphabet V = {a, b, c, d, e}, it is represented as... Editing operations include deletion, replacement, and insertion, represented by formulas 7, 8, and 9 respectively:

[0025]

[0026] Edit script Defined as a series of editing operations δ1,…,δ T Application in trees The above is represented as: in Operation is The final tree edit distance is:

[0027]

[0028] When editing fails due to an inappropriate location of the editing operation, the cost c(δ) i c(δ, x, y) = 0; when the edit operation contains a keyword node, c(δ) = 0. i (x, y) = 2; For normal editing operations, c(δ) i (x,u)=1, and finally, the AST tree edit distance is expressed as Equation 11:

[0029]

[0030] in The total cost of editing the script is expressed as:

[0031]

[0032] Finally, the positive reward function R + =CodeBLEU + ε·T5Score + TED AsTSince the T5Score score is between 0 and 1, ε is used to compensate for the proportion of T5Score in CodeBLEU, ε = 50.

[0033] 1.2) Calculate the penalty term P KL ;

[0034] The method for calculating the penalty is as follows:

[0035] The penalty term P is used to penalize cases where the Actor's output distribution deviates too far from the Reference model. The calculation of the penalty term uses the KL divergence, an asymmetric measure of the difference between two probability distributions, as shown in Formula 13.

[0036]

[0037] Among them, B code It is a vulnerable function, and Rep is the generated patch. It is the probability of each patch token generated by the Actor model, π. Ref (Rep|B code Similarly, in the correspondence between vulnerability repair and reinforcement learning, B... code The initial state is B. As the policy is chosen, the state changes to B. code With the continuous combination of patching and repair, the action space of the strategy is the vocabulary.

[0038] 1.3) Calculate the comprehensive reward function R φ The formula for the comprehensive reward function is:

[0039] R φ =R + -P KL

[0040] Step 2: Create a Critic model, using MSE error as the loss function to update the model. The formula is:

[0041]

[0042] in, For the target state value of the Critic model, the time difference (TD) error is used. Use the comprehensive reward function R at time t φ,t and the old state value v at time t+1 old (s t+1 The calculation is performed using γ, where γ is the conversion factor with a value of 0.95, and v new (s t ) represents the new predicted state value at time t;

[0043] The loss function of the Critic model is:

[0044] The Critic model is derived from the encoder part of the SASTCodeT5 model, with the input being the state s at time t. t , by vulnerable code B code Composed of a repair patch (Rep), it outputs the value v(s) of each token in the current state after passing through a fully connected layer. For the Critic model, the MSE error is used as the loss function to update the model, as shown in Equation 4-15:

[0045]

[0046] in, For the target value of the Critic model, for the target value at time t, the value at time t+1 when the experience was collected is used, through time difference error. old (s t+1 The target value at time t is approximated by the sum of the current reward value at time t and the reward value at time t, because the current real reward has already been calculated at time t. If the prediction is divided into two parts, the prediction for the current time and the prediction for the future time, when the real value of the prediction for the current time is known, the prediction for the current time is replaced with the real value of the current time to reduce the error of the total prediction.

[0047]

[0048] Therefore, Formula 15 becomes:

[0049]

[0050] Step 3: Create the Actor model. The Actor model is a stacked model based on the transformer architecture, using the CodeT5 model proposed by Google. The formula is:

[0051]

[0052] Where φ represents the parameters of the Actor model, ∈ is a hyperparameter controlling the clipping amplitude, usually taken as 0.15, and clip(a,b,c) means restricting a to the interval [b,c]; A t The min(·,·) operation represents the dominance function at time t.

[0053] The loss function for the Actor model is:

[0054] Based on the policy gradient algorithm, the training objective of the Actor model is to maximize the expected value of the state value. The state value is calculated by multiplying the probability distribution of the policy output at time t by the action state value at the current time.

[0055]

[0056] For action state values, if calculated according to the above formula, a positive value will be obtained regardless of the action taken. As a result, the Actor model will continuously update in this direction. Therefore, a baseline standard is used as a measure for the action value function. Values ​​above this baseline are positive, and values ​​below this baseline are negative, ensuring normal training of the model. The value function after incorporating the baseline is called the dominance function.

[0057] A t =Q(s) t ,a t )-V(s t (19)

[0058] The loss function of the Actor model becomes:

[0059]

[0060] In PPO, after importance sampling, the on-policy strategy is changed to an off-policy strategy. Under the on-policy approach, one trajectory is sampled each time. After updating the Actor, the parameters change, and the sampled data can no longer be used, requiring resampling. Under the off-policy approach, a different strategy is used to interact with the environment for sampling, and this data is used to update the Actor's parameters. Thus, the Actor is updated multiple times with a batch of data. The core of importance sampling is as follows:

[0061]

[0062] The process of finding the expectation of a function f(x) on the distribution p(x) is transformed into finding... The expectation on the distribution q(x), combined with formula 22, transforms formula 20 into formula 23:

[0063]

[0064] The loss in Equation 23 is further optimized using the PPO-Clip method:

[0065]

[0066] in, ∈ is a hyperparameter that controls the clipping range. clip(a,b,c) means restricting a to the interval [b,c].

[0067] Step 4: Utilize Experiences and learning from interacting with the environment:

[0068] 4.1) Data Acquisition Process: The vulnerable code generates a patch through the Actor in inference mode and merges it with the source code through patch location. Then, the reward value is given by the comprehensive reward function in step 1. The vulnerable code and the patch are concatenated into a simple sequence and input into the model and Reference model in steps 2 and 3 to obtain the Actor logits output of the Actor model, the state value of the Critic model, and the Ref logits output of the Reference model, which constitute a complete trajectory experience.

[0069] 4.2) Learning process:

[0070] Using the collected trajectory experience, online reinforcement learning is performed. The model is trained for 100 epochs using the backpropagation algorithm, with a batch size of 1 for each epoch. After training for 100 epochs, the final Actor model is obtained, which is the final repair model.

[0071] The beneficial effects of this invention are as follows: by integrating reinforcement learning and pre-trained models, the syntactic accuracy and semantic matching degree of patches generated by vulnerability repair models are significantly improved. Its core advantages are reflected in: a comprehensive reward function built based on CodeBLEU, T5Score, and Tree Edit Distance (TED). This function not only captures the semantic similarity of high-dimensional word vectors output by the pre-trained encoder through T5Score, but also innovatively introduces adjustable cost weights in TED to accurately quantify the structural differences between the generated AST and the reference AST, thus strengthening syntactic alignment from multiple dimensions. Simultaneously, the KL divergence penalty term constrains the deviation between the Actor output distribution and the pre-trained Reference model. Combined with a Critic network built based on the vulnerability repair model encoder, stable and efficient policy optimization is achieved within the PPO and Advantage Actor-Critic framework. Ultimately, while ensuring the functional correctness of the generated code, the structural similarity and code standardization of the patches and real-world repair solutions are significantly improved. Attached Figure Description

[0072] Figure 1 This is a training framework diagram of the present invention.

[0073] Figure 2 Graph of the comprehensive reward function;

[0074] Figure 3 Here is a structural diagram of the Critic model;

[0075] Figure 4 Flowchart of reinforcement learning algorithm;

[0076] Figure 5 A diagram illustrating the process of collecting experience from the interaction between the Actor and Critic models and the environment. Detailed Implementation

[0077] A reinforcement learning training method to improve the code structure awareness capability of vulnerability remediation models:

[0078] Step 1: Design a multi-angle structural similarity reward function, and construct the comprehensive reward function as follows: Figure 2 As shown:

[0079] 1.1) Calculate the positive reward function R + ;

[0080] In software engineering, evaluating the quality of patch remediation requires a rigorous set of standards. The most important standard is that it should not introduce new vulnerabilities while thoroughly resolving the existing ones. Automated patch quality evaluation is a complex and laborious task, as it involves merging the patch into the appropriate part of the source code and reproducing the attack in a specific project and hardware environment. While merging patches into a project can be automated, reproducing the attack in a specific hardware and software environment is extremely resource-intensive and sometimes impossible. The most direct approach is manual verification by the project's programmers, but this is impractical for large-scale deep learning. In deep learning evaluation methods, possible approaches include checking whether the string perfectly matches the actual label or whether the merged code structure is highly similar to the correct code structure to assess the quality of the generated patch. Therefore, this study focuses on using various NLP task metrics to design a reward function that can accurately evaluate patch quality from multiple dimensions.

[0081] In NLP model evaluation, various metrics are widely used to measure model performance, including N-gram-based metrics, edit distance-based metrics, and embedding-based metrics. These metrics differ in definition, focus, and calculation methods, providing multi-faceted considerations for NLP model performance evaluation. Among N-gram-based metrics...

[0082] CodeBLEU is designed specifically for code, incorporating BLEU while also considering syntactic and semantic structure matching. Therefore, CodeBLEU can be used in the reward function. Among embedding-based metrics, MEANT2.0 and YISI-1 use relatively simple similarity calculations, while BertScore uses contextual embeddings for calculation and improves the metric's effectiveness through greedy matching and weighted importance. However, this invention has made appropriate modifications to BertScore to suit the task of this study.

[0083] Edit distance-based metrics quantify similarity by calculating the number of edit operations required to transform a candidate into a reference. This invention does not use these traditional string edit distances but introduces a tree edit distance. Tree edit distance describes the minimum number of node deletions, insertions, and replacements required to transform from one tree to another. This differs somewhat from the metric in CodeBLEU that measures the number of matching subtrees in an AST. This invention argues that for two ASTs, the smaller the edit distance from one to the other, the higher the similarity between the two ASTs. Therefore, this invention incorporates tree edit distance into the reward function calculation. In summary, this invention considers CodeBLEU, T5Score, and tree edit distance when designing the positive reward function.

[0084] 1.1.1) CodeBLEU

[0085] CodeBLEU=α·BLEU+β·BLEU weight +γ·Match ast +δ·Match df (1)

[0086] Where α, β, γ, and δ are weighting coefficients, and BLEU represents the standard BLEU and BLEU. weight For weighted BLEU, Match ast Matching percentage of AST subtrees df The proportion of data stream matching;

[0087] 1.1.2) T5Score: BertScore improves upon two common shortcomings of N-gram-based metrics: first, N-gram-based metrics generally cannot match sentence semantics; second, N-gram models cannot capture long-range dependencies. BertScore uses tokenized embeddings containing contextual semantics to calculate similarity because contextual embeddings are trained to effectively capture long-range dependencies and order. However, this invention only utilizes the main method of BertScore and does not choose to use the Bert model. Instead, it uses the pre-trained SASTCodeT5 encoder model, which has been trained on a large-scale code corpus relative to the Bert model and has undergone secondary training on error and vulnerability datasets. Its word embedding representation capability for code is far superior to that of the Bert model. This invention calls this metric T5Score, and its calculation method consists of the following steps:

[0088] 1.1.2.1) Token representation: Assuming the synthesized patch code after generation is represented as x = {x1, x2, ..., x} k}, the reference code is Using the SASTCodeT5 encoder, x and... Generate context embeddings x = {x1, x2, ..., x} k}and

[0089] 1.1.2.2) Calculate similarity: Calculate the similarity between the context embeddings of the vector representations, x i and The cosine similarity is By normalizing, the calculation is simplified to an inner product.

[0090] 1.1.2.3) Calculate precision and recall: The full T5Score uses greedy matching to match each token in x with... Token matching in the middle is used to calculate recall, and Each token in the string is matched against one of the tokens in the string to calculate precision, and each token is matched against the most similar token in another string; then BLEU in CodeBLEU is used. weight The weighting method w(x) i This means setting the keyword weight to 2.5 times that of other keywords, with precision and recall rates as shown in the formulas:

[0091]

[0092] Among them, R T5 P represents recall rate. T5 Indicate precision;

[0093] 1.1.2.4) Scaling: To scale the cosine similarity from -1 to 1 to between 0 and 1, calculate R from the previous step. T5 P T5 The average value, let R be. T5 The average value is a, P t5 The average value is b, which, after scaling, is:

[0094]

[0095] 1.1.2.5) Calculate T5Score

[0096]

[0097] 1.1.3) Tree Edit Distance (TED):

[0098] Tree Edit Distance (TED) is calculated using dynamic programming to determine the minimum number of node deletions, insertions, and replacements required to move from one tree to another. This invention introduces TED into AST evaluation, customizing the distance calculation method and cost within the algorithm.

[0099] In TED, trees and forests, edit operations and edit scripts, and tree edit distances are defined. For a tree on the alphabet V = {a, b, c, d, e}, it is represented as... Editing operations include deletion, replacement, and insertion, represented by formulas 7, 8, and 9 respectively:

[0100]

[0101] Edit script Defined as a series of editing operations δ1,…,δ T Application in trees The above is represented as: in Operation is The final tree edit distance is:

[0102]

[0103] When editing fails due to an inappropriate location of the editing operation, the cost c(δ) i c(δ, x, y) = 0; when the edit operation includes a keyword node, c(δ) = 0. i (x, y) = 2; For normal editing operations, c(δ) i (x, y) = 1, and finally, the AST tree edit distance is expressed as Equation 11:

[0104]

[0105] in The total cost of editing the script is expressed as:

[0106]

[0107] Finally, the positive reward function R + =CodeBLEU + ε·T5Score + TED AsT Since the T5Score score is between 0 and 1, ε is used to compensate for the proportion of T5Score in CodeBLEU, ε = 50.

[0108] 1.2) Calculate the penalty term P KL ;

[0109] The method for calculating the penalty is as follows:

[0110] The penalty term P is used to penalize situations where the Actor's output distribution deviates too far from the Reference model. The calculation of the penalty term uses the KL divergence, an asymmetric measure of the difference between two probability distributions, as shown in Formula 13.

[0111]

[0112] Among them, B code It is a vulnerable function, and Rep is the generated patch. It is the probability of each patch token generated by the Actor model, π. Ref (Rep|B code Similarly, in the correspondence between vulnerability repair and reinforcement learning, B... code The initial state is B. As the policy is chosen, the state changes to B. code With the continuous combination of patching and repair, the action space of the strategy is the vocabulary.

[0113] 1.3) Calculate the comprehensive reward function R φ The formula for the comprehensive reward function is:

[0114] R φ =R + -P KL

[0115] Step 2: Create a Critic model, such as... Figure 3 As shown, the MSE error is used as the loss function to update the model, and the formula is:

[0116]

[0117] in, The target state value of the Critic model is obtained through the Temporal Difference (TD) error. Use the comprehensive reward function R at time t φ,t and the old state value v at time t+1 old (s t+1 The calculation is performed using γ, where γ is the conversion factor with a value of 0.95, and v new (s t ) represents the new predicted state value at time t;

[0118] The loss function of the Critic model is:

[0119] The Critic model is derived from the encoder part of the SASTCodeT5 model, with the input being the state s at time t. t , by vulnerable code B codeComposed of a repair patch (Rep), it outputs the value v(s) of each token in the current state after passing through a fully connected layer. For the Critic model, the MSE error is used as the loss function to update the model, as shown in Equation 4-15:

[0120]

[0121] in, For the target value of the Critic model, for the target value at time t, the value at time t+1 when the experience was collected is used, through time difference error. old (s t+1 The target value at time t is approximated by the sum of the current reward value at time t and the reward value at time t, because the current real reward has already been calculated at time t. If the prediction is divided into two parts, the prediction for the current time and the prediction for the future time, when the real value of the prediction for the current time is known, the prediction for the current time is replaced with the real value of the current time to reduce the error of the total prediction.

[0122]

[0123] Therefore, Formula 15 becomes:

[0124]

[0125] Step 3: Create the Actor model. The Actor model is a stacked model based on the transformer architecture, using the CodeT5 model proposed by Google. The formula is:

[0126]

[0127] Where φ represents the parameters of the Actor model, ∈ is a hyperparameter controlling the clipping amplitude, usually taken as 0.15, and clip(a,b,c) means restricting a to the interval [b,c]; A t The min(·,·) operation represents the dominance function at time t.

[0128] The loss function for the Actor model is:

[0129] Based on the policy gradient algorithm, the training objective of the Actor model is to maximize the expected value of the state value. The state value is calculated by multiplying the probability distribution of the policy output at time t by the action state value at the current time.

[0130]

[0131] For action state values, if calculated according to the above formula, a positive value will be obtained regardless of the action taken. As a result, the Actor model will continuously update in this direction. Therefore, a baseline standard is used as a measure for the action value function. Values ​​above this baseline are positive, and values ​​below this baseline are negative, ensuring normal training of the model. The value function after incorporating the baseline is called the dominance function.

[0132] A t =Q(s) t ,a t )-V(s t (19)

[0133] The loss function of the Actor model becomes:

[0134]

[0135] In PPO, after importance sampling, the on-policy strategy is changed to an off-policy strategy. Under the on-policy approach, one trajectory is sampled each time. After updating the Actor, the parameters change, and the sampled data can no longer be used, requiring resampling. Under the off-policy approach, a different strategy is used to interact with the environment for sampling, and this data is used to update the Actor's parameters. Thus, the Actor is updated multiple times with a batch of data. The core of importance sampling is as follows:

[0136]

[0137] The process of finding the expectation of a function f(x) on the distribution p(x) is transformed into finding... The expectation on the distribution q(x), combined with formula 22, transforms formula 20 into formula 23:

[0138]

[0139] The loss in Equation 23 is further optimized using the PPO-Clip method:

[0140]

[0141] in, ∈ is a hyperparameter that controls the clipping range. clip(a,b,c) means restricting a to the interval [b,c].

[0142] Step 4: Using π φold The process of acquiring experience and learning through interaction with the environment, as described by Actor and Critic models, is as follows: Figure 5 As shown:

[0143] 4.1) Data Acquisition Process: The vulnerable code generates a patch through the Actor in inference mode and merges it with the source code through patch location. Then, the reward value is given by the comprehensive reward function in step 1. The vulnerable code and the patch are concatenated into a simple sequence and input into the model and Reference model in steps 2 and 3 to obtain the Actor logits output of the Actor model, the state value of the Critic model, and the Ref logits output of the Reference model, which constitute a complete trajectory experience.

[0144] 4.2) Learning process:

[0145] Utilizing the collected trajectory experience, online reinforcement learning is performed. The model is trained for 100 epochs using the backpropagation algorithm, with a batch size of 1 per epoch. After 100 epochs, the resulting Actor model serves as the final repair model. The specific method is as follows... Figure 4 As shown.

Claims

1. A reinforcement learning training method to improve the code structure awareness capability of a vulnerability remediation model, characterized in that: Step 1) Design a multi-angle structural similarity reward function: Step 1.1) Calculate the positive reward function The calculation method is as follows: Step 1.1.1) CodeBLEU Where α, β, γ, and δ are weighting coefficients, and BLEU represents the standard BLEU. For weighted BLEU, The proportion of matching subtrees in the AST. The proportion of data stream matching; Step 1.1.2) T5Score Step 1.1.2.1) Token representation: Assuming the synthesized patch code is generated after the patch is created, it is represented as follows: The reference code is Using encoders respectively and Generate context embeddings and ; Step 1.1.2.2) Calculate similarity: Calculate the similarity between the context embeddings of the vector representations. and The cosine similarity is By normalizing, the calculation is simplified to an inner product. ; Step 1.1.2.3) Calculate precision and recall: The complete T5Score uses greedy matching to... Each token in Token matching in the middle is used to calculate recall, and Each token in One token is matched against another token in the sentence to calculate precision; each token is matched against the most similar token in the other sentence. Then, CodeBLEU's [text / framework / data] is used. Weighting method This means setting the keyword weight to 2.5 times that of other keywords, with precision and recall rates as shown in the formulas: in, Indicates recall rate, Indicate precision; Step 1.1.2.4) Scaling: To scale the cosine similarity from -1 to 1 to between 0 and 1, calculate the similarity from the previous step... , The average value, let The average value is a. The average value is b, which, after scaling, is: Step 1.1.2.5) Calculate T5Score Step 1.1.3) Tree edit distance TED Dynamic programming is used to calculate the minimum number of node deletions, insertions, and replacements required to move from one tree to another, thus introducing TED (Thinking-Based Tree) into the evaluation of Abstract Syntax Trees (ASTs). In TED, trees and forests, edit operations and edit scripts, and tree edit distances are defined for the alphabet. A tree on the ground, represented as Editing operations include deletion, replacement, and insertion, represented by formulas 7, 8, and 9 respectively: Edit script Defined as a series of editing operations Application in trees The above is represented as: ,in Operation is The final tree edit distance is: When editing fails due to an inappropriate location, the cost is... When the editing operation contains a keyword node, Normal editing operations, Finally, the AST tree edit distance is expressed as Equation 11: in The total cost of editing the script is expressed as: Ultimately, the positive reward function ,because The score is between 0 and 1, using right The proportion in CodeBLEU will be compensated. =50; Step 1.2) Calculate the penalty term ; Step 1.3) Calculate the comprehensive reward function The formula for the comprehensive reward function is: Step 2) Create the Critic model, using the MSE error as the loss function to update the model. The formula is: in, For the target state value of the Critic model, the time difference (TD) error is used. Use the comprehensive reward function at time t and the old state value at time t+1 To calculate, This is a conversion factor with a value of 0.

95. The new predicted state value at time t; Step 3) Create the Actor model. The Actor model is a stacked model based on the transformer architecture, using the CodeT5 model, with the following formula: in, The parameters of the Actor model are represented. This is a hyperparameter that controls the cutting range; it is set to 0.

15. This means restricting 'a' to the interval [b, c]. This represents the dominance function at time t. Minimum value operation; ; Step 4) Utilize Experiences and learning from interacting with the environment: Step 4.1) Data Acquisition Process: The vulnerable code generates a repair patch through the Actor in inference mode and merges it with the source code through patch location. Then, the reward value is given by the comprehensive reward function in Step 1. The vulnerable code and the repair patch are concatenated into a simple sequence and input into the model and Reference model in Steps 2 and 3 to obtain the Actor logits output of the Actor model, the state value of the Critic model, and the Ref logits output of the Reference model, which constitute a complete trajectory experience. Step 4.2) Learning Process: Using the collected trajectory experience, online reinforcement learning is performed. The model is trained for 100 epochs using the backpropagation algorithm, with a batch size of 1 for each epoch. After training for 100 epochs, the final Actor model is obtained, which is the final repair model.

2. The reinforcement learning training method for improving the code structure awareness capability of a vulnerability remediation model according to claim 1, characterized in that, The method for calculating the penalty term in step 1.2) is as follows: Penalty items To penalize cases where the Actor's output distribution deviates too far from the Reference model, the penalty term is calculated using KL divergence, an asymmetric measure of the difference between two probability distributions, as shown in formula 13. in, It is a vulnerable function. It is the generated fix patch. It represents the probability of each patch token generated by the Actor model. Similarly, in the correspondence between vulnerability repair and reinforcement learning, The initial state is [state 1], and as the policy is chosen, the state changes to [state 2]. With the continuous combination of patching and repair, the action space of the strategy is the vocabulary.

3. The reinforcement learning training method for improving the code structure awareness capability of a vulnerability remediation model according to claim 1, characterized in that, In step 2), the loss function of the Critic model is: The Critic model originates from the model encoder part, with the input being the state t at time t. 𝑡 From vulnerable code With the fix patch The model is composed of a fully connected layer that outputs the value v(t) of each token in the current state. For the Critic model, the MSE error is used as the loss function to update the model, as shown in Equation 4-15. in, For the target value of the Critic model, for the target value at time t, the value at time t+1 of the data collection time is used through time difference error. The target value at time t is approximated by the sum of the reward value at time t and the reward value at time t, because the actual reward at time t has already been calculated. If the prediction is divided into two parts, the prediction for the current time and the prediction for the future time, when the actual value of the prediction for the current time is known, the prediction for the current time is replaced with the actual value of the current time, thereby reducing the error of the total prediction. Therefore, Formula 15 becomes: 。 4. The reinforcement learning training method for improving the code structure awareness capability of a vulnerability remediation model according to claim 1, characterized in that, In step 3), the loss function of the Actor model is: Based on the policy gradient algorithm, the training objective of the Actor model is to maximize the expected value of the state value. The state value is calculated by multiplying the probability distribution of the policy output at time t by the action state value at the current time. For action state values, if calculated according to the above formula, a positive value will be obtained regardless of the action taken. As a result, the Actor model will continuously update in this direction. Therefore, a baseline standard is used as a measure for the action value function. Values ​​above this baseline are positive, and values ​​below this baseline are negative, ensuring normal training of the model. The value function after incorporating the baseline is called the dominance function. The loss function of the Actor model becomes: In PPO, after importance sampling, the on-policy strategy is changed to an off-policy strategy. Under the on-policy approach, one trajectory is sampled each time. After updating the Actor, the parameters change, and the sampled data can no longer be used, requiring resampling. Under the off-policy approach, a different strategy is used to interact with the environment for sampling, and this data is used to update the Actor's parameters. Thus, the Actor is updated multiple times with a batch of data. The core of importance sampling is as follows: Soon to seek Functions on distribution The expectation is transformed into a demand In distribution The expected value above, combined with formula 22, transforms formula 20 into formula 23: The loss in Equation 23 is further optimized using the PPO-Clip method: in, , It is a hyperparameter that controls the cutting range. This means restricting a to the interval [b, c].

Images