Code model robustness evaluation method and system based on variable importance analysis

Abstract

The present invention provides a method and system for evaluating the robustness of a code model based on variable importance analysis. The method includes: obtaining a code model to be evaluated and a source code sample, extracting variables from a real source code library to construct a first set of replacement variables; calculating the importance score of each initial variable in the source code sample; screening replacement variables for important variables in the first set of replacement variables to construct a second set of replacement variables, and applying length constraints and K-means clustering in the screening process to ensure semantic diversity; using an improved beam search method to perform variable replacement testing, and expanding the search space by merging the results of previous and subsequent iterations to avoid falling into local optimality. The present invention can systematically and efficiently evaluate the sensitivity of a code model to changes in variable naming, accurately identify model robustness weaknesses even under limited query conditions, and provide technical support for improving the stability and security of the code model.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a code model robustness evaluation method and system based on variable importance analysis. Background Art

[0002] With the development of deep learning technology, various code models such as CodeBERT, GraphCodeBERT, and CodeT5 have been widely used in tasks such as code comprehension, classification, and generation. However, research has shown that these code models are often very sensitive to small perturbations in the input. In particular, they lack sufficient robustness to code transformations that preserve semantics but change form, such as variable renaming. For example, simply renaming certain variables in the source code can lead to completely different model outputs, seriously affecting the reliability and security of code models in practical applications.

[0003] Currently, there are several main methods for evaluating the robustness of code models:

[0004] 1. Random selection methods (such as MHM) randomly select 1,500 words from a fixed dictionary as replacement variables. However, this method may result in too long variable names and limited diversity of replacement variables.

[0005] 2. Methods based on pre-trained model generation (such as ALERT) generate 60 alternative variables for each original variable through a pre-trained mask model, but these alternative variables often lack diversity;

[0006] 3. Methods based on historical attack data (such as RNNS) use historical attack data as search seeds to identify potential adversarial variable replacements, but each variable name is limited to 60 variable names.

[0007] These methods typically employ greedy strategies or a combination of greedy and heuristic algorithms to search for adversarial examples. While effective, these algorithms are prone to falling into local optima, leading to excessive perturbations and low attack success rates under limited query budgets. Furthermore, existing methods fail to fully exploit the diversity of alternative variable sets, fail to consider the differences in the importance of variables to model output, and fail to effectively address the issue of expanding the search space.

[0008] Therefore, there is an urgent need for a method that can comprehensively evaluate the robustness of code models. This method should be able to identify the importance of variables, provide a diverse selection of alternative variables, and improve the search strategy to avoid local optimal solutions, thereby achieving efficient robustness evaluation under a limited query budget. Summary of the Invention

[0009] This paper provides a code model robustness assessment method based on variable importance analysis, addressing existing technical issues such as insufficient diversity of alternative variables, the tendency of search algorithms to fall into local optimal solutions, and low query efficiency. By calculating variable importance, using a constrained K-means clustering algorithm to identify diverse alternative variables in the embedding space, and employing an improved beam search algorithm to construct test samples, the present invention significantly improves the efficiency and accuracy of code model robustness assessment.

[0010] In order to achieve the above-mentioned object of the invention, the technical solution provided by the present invention includes:

[0011] A code model robustness evaluation method based on variable importance analysis includes the following steps:

[0012] S1. Obtain the code model and source code sample to be evaluated; extract variables from the real source code library to construct a first set of replacement variables;

[0013] S2. Calculate the importance score of each initial variable in the source code sample to the output of the code model to be evaluated, wherein the importance score is calculated based on the soft label method;

[0014] S3. Filtering the first set of replacement variables for the initial variables whose importance scores are greater than a first preset threshold value to construct a second set of replacement variables;

[0015] S4. Iteratively perform variable replacement on the source code sample using the second set of replacement variables, generate and evaluate test samples in each iteration, and continuously optimize the test samples based on the difference in prediction results between the test samples in each round and the source code samples of the code model to be evaluated until the robustness evaluation is completed.

[0016] Preferably, the method for constructing the second set of substitution variables includes:

[0017] S301. From the first set of replacement variables, extract variables whose lengths differ from the initial variable lengths whose importance scores are greater than the first preset threshold and are within a second preset threshold range, and construct a third set of replacement variables;

[0018] S302. Obtain an embedded vector representation of each variable in the third replacement variable set, and apply a K-means clustering algorithm to cluster semantically similar variables in the third replacement variable set into K candidate clusters;

[0019] S303. Select at least one variable in each candidate cluster to form a second replacement variable set.

[0020] Preferably, step S4 includes:

[0021] S401 initializes the test sample set, adding the source code sample to the test sample set;

[0022] S402. Traverse each sample in the test sample set, for the initial variable currently being processed, replace the initial variable with the candidate replacement variable in the second replacement variable set, generate a new test sample and add it to the temporary sample set;

[0023] S403. Test each sample in the temporary sample set using the code model to be evaluated, and calculate the difference between the predicted probability distribution of each sample and the source code sample;

[0024] S404. Combine the test sample set retained from the previous iteration with the currently generated temporary sample set, and select the top B samples with the largest difference values ​​as the new test sample set;

[0025] S405. Repeat steps S402 to S404 until the stop condition is met.

[0026] Preferably, the method for selecting the initial variables of the current process in step S402 includes:

[0027] Initial variables with importance scores greater than a first preset threshold are extracted and sorted in descending order of importance scores. In each iteration, they are selected in sequence as initial variables for current processing.

[0028] Preferably, the stopping condition includes:

[0029] Discover test samples that can change the predictions of the code model under evaluation;

[0030] All initial variables with importance scores greater than the first preset threshold have been processed;

[0031] The number of iterations reaches a third preset threshold.

[0032] Preferably, the method for calculating the importance score of each initial variable in the source code sample to the output result of the code model to be evaluated in step S2 includes: replacing the initial variables to be calculated in all instances of the source code sample with special tags to ensure the compilability of the source code sample and comprehensively calculate the importance of the initial variables to be calculated.

[0033] Preferably, when applying the K-means clustering algorithm to cluster the third set of replacement variables in step S302, the Euclidean distance is used as the similarity metric between the embedding vectors of each variable.

[0034] The present invention also provides a code model robustness evaluation system based on variable importance analysis, which is used to implement the above-mentioned code model robustness evaluation method based on variable importance analysis.

[0035] Beneficial effects

[0036] 1. By introducing a variable importance calculation method based on soft labels, we can accurately identify the variables that have the greatest impact on model output, effectively reducing the number of variables to be processed and improving evaluation efficiency. Specifically, this method quantifies the importance of variables by calculating the change in the model's predicted probability distribution before and after variable replacement, ensuring that the most influential variables are processed first.

[0037] 2. By processing the set of alternative variables using a constrained K-means clustering algorithm, we effectively group semantically similar variables, significantly increasing the diversity of alternative variables. This method first applies length constraints to filter alternative variables. Then, we measure the similarity between variables using the Euclidean distance of their embedding vectors. Representative variables are selected from each cluster, forming a more diverse set of alternative variables and providing a richer set of candidate variables for subsequent searches.

[0038] 3. An improved beam search algorithm is used to generate test samples, effectively avoiding the problem of being trapped in a local optimum. Unlike traditional greedy algorithms, the beam search method of this invention merges the samples retained from the previous iteration with the currently generated samples in each iteration and selects the top B samples with the largest difference values ​​as the new test sample set. This significantly expands the search space and increases the probability of finding the optimal test sample.

[0039] 4. Under a limited query budget (e.g., 100 queries), the proposed method can achieve a higher attack success rate and a lower perturbation rate. Experimental results show that, under the same query budget, the proposed method significantly outperforms existing soft label attack methods, increasing the average attack success rate by 2.45% and reducing the average perturbation rate by 2.33%.

[0040] 5. The method of the present invention has wide applicability and is applicable to multiple programming languages ​​(such as Java, Python, and C) and multiple code-related tasks (such as author prediction, clone detection, vulnerability detection, and problem-solving classification). It shows good evaluation results for different code models (such as CodeBERT, GraphCodeBERT, and CodeT5). BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The present invention is a flowchart of a method for evaluating the robustness of a code model based on variable importance analysis in a preferred embodiment disclosed in the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inner", "outer", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or components referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0043] Example

[0044] like Figure 1 As shown, the present invention discloses a code model robustness evaluation method based on variable importance analysis, comprising the steps of:

[0045] S1. Obtain the code model to be evaluated and the source code sample; extract variables from the real source code library to construct the first replacement variable set.

[0046] Specifically, obtaining appropriate evaluation objects and data is a prerequisite for the evaluation process, and constructing a high-quality set of alternative variables directly affects the effectiveness of variable replacement. The present invention first determines the code model to be evaluated, which can be a pre-trained code understanding model or a dedicated code task model, used to perform tasks such as code clone detection, vulnerability identification, and code summary generation. Source code samples are collected from multiple channels. These samples need to cover mainstream programming languages ​​and should have functional integrity and syntactic correctness. The method of the present invention pays special attention to the task relevance of source code samples, that is, the selected samples should match the specific task type of the model to be evaluated, such as selecting code snippets containing known security vulnerabilities for vulnerability detection models.

[0047] The method uses a specialized code parser to perform syntactic analysis on the collected source code samples, extracting various variables, including local variables, global variables, function parameters, and class attributes. The extracted variable information is structured to record attributes such as each variable's name, type, scope, and frequency of occurrence. To ensure the quality of the first replacement variable set, the method systematically cleans the variable set, including removing duplicate variable names, filtering variable names of inappropriate length, processing special characters, and filtering variable names that overlap with reserved words in the programming language.

[0048] By optimizing the distribution of variables across different categories, introducing domain-specific variable names, and weighting them based on frequency of use, the proposed method ultimately constructs a structured, semantically rich first set of replacement variables. This set of replacement variables not only incorporates the original variable names extracted from real code but also undergoes systematic cleaning and optimization. This provides a high-quality candidate library for subsequent variable replacement strategies, ensuring that the generated code variants maintain both syntactic correctness and sufficient semantic rationality and diversity, effectively supporting robustness assessment of code models.

[0049] S2. Calculate the importance score of each initial variable in the source code sample to the output result of the code model to be evaluated, wherein the importance score is calculated based on a soft label method.

[0050] Variable importance calculation aims to quantify the impact of each variable on model decisions, providing a scientific basis for subsequent variable replacement strategies. This paper uses a soft-label approach to calculate variable importance. This approach not only considers the model's final predicted category (hard label) but also fully utilizes the complete probability distribution information (soft label) output by the model, thereby accurately measuring the variable's impact.

[0051] In some preferred embodiments, a specific importance score calculation process is provided, which specifically includes:

[0052] Assume that the source code sample is X, which contains n initial variables {var_1, var_2, ..., var_n}, the code model to be evaluated is F, and the predicted probability distribution for input X is P(Y|X), where Y={y_1, y_2, ..., y_m} represents m possible output categories. For each initial variable var_i, first generate a modified code sample X / {var_i}, where the variable var_i is replaced with a specific special token (such as "UNK" or " <mask>”) This step is to ensure the compilability of the source code sample and comprehensively calculate the importance of the initial variables to be calculated. Then, the original sample X and the modified sample X / {var_i} are respectively input into the code model F to obtain the corresponding predicted probability distributions P(Y|X) and P(Y|X / {var_i}).

[0053] The importance calculation method of the present invention distinguishes between two cases: when the model's predicted label remains unchanged after variable substitution and when the predicted label changes. Suppose the true label of the original sample X is y_true, and the predicted label of model F for X is also y_true, that is, F(X) = y_true. When the model's predicted label remains y_true after substitution of variable var_i, that is, F(X / {var_i}) = y_true, the importance score I(var_i) of variable var_i is calculated as follows:

[0054] I(var_i) = P(y_true|X) - P(y_true|X / {var_i})

[0055] This formula quantifies the decrease in the probability of the model predicting the correct category due to replacing the variable var_i. The greater the decrease in probability, the more significant the impact of the variable on the model's prediction, that is, the higher its importance.

[0056] When the variable var_i is replaced, the model prediction label becomes y_other, that is, F(X / {var_i}) = y_other ≠ y_true, and the importance score of variable var_i I(var_i) is calculated by the following formula:

[0057] I(var_i) = [P(y_true|X) - P(y_true|X / {var_i})] - [P(y_other|X / {var_i}) - P(y_other|X)]

[0058] This formula not only considers the decrease in the probability of the correct category but also the increase in the probability of the incorrect category, thus more comprehensively reflecting the impact of variable replacement on the model's decision boundary. The first term represents the decrease in the probability of the correct category, and the second term represents the increase in the probability of the incorrect category. The difference between the two comprehensively reflects the importance of the variable.

[0059] It should be understood that the soft labeling method of the present invention offers significant advantages over traditional hard labeling methods (which only consider whether the predicted category has changed). The soft labeling method can capture variables that, while not changing the final predicted category, significantly impact the prediction confidence, providing a more granular assessment of importance. Especially in the case of high-confidence predictions, hard labeling methods may struggle to distinguish the relative importance of variables. By considering changes in probability distributions, the soft labeling method can more accurately identify key variables.

[0060] Through this importance calculation, the method of the present invention can sort all original variables in the source code in descending order of importance, forming a variable importance ranking list. This ranking list will be used to guide variable replacement strategies in subsequent steps, prioritizing variables with higher importance scores, thereby improving evaluation efficiency and attack success rate. Furthermore, variable importance analysis provides valuable insights into the interpretability of code models, helping to understand the model's decision-making mechanisms and potential robustness flaws.

[0061] S3. Filtering the first replacement variable set to include replacement variables for the initial variables whose importance scores are greater than a first preset threshold value to construct a second replacement variable set.

[0062] This step improves the pertinence and efficiency of subsequent variable replacement by focusing on variables that significantly influence the model output. The present invention first sets a first preset threshold value θ to determine the significance level of variable importance. This threshold value can be determined through empirical setting, data distribution analysis, or adaptive calculation. For example, all variable importance scores calculated in step S2 are sorted in descending order, and the top 10%-30% of variable importance values ​​are selected as the threshold value, or the mean plus n times the standard deviation is used as the threshold value.

[0063] In the actual implementation process, for each initial variable var_i whose importance score I(var_i) is greater than the threshold θ, the method of the present invention selects a suitable replacement variable from the first replacement variable set. The selection process takes into account multi-dimensional matching characteristics, including variable type compatibility, naming style consistency, semantic relevance, and context adaptability. For type compatibility, ensure that the replacement variable is compatible with the original variable at the language syntax level, such as integer variables should be replaced with integer variable names; for naming style, keep the naming conventions consistent before and after replacement, such as camel case naming or snake case naming; for semantic relevance, give priority to replacements that are semantically close to the original variable but not completely equivalent, maintain the rationality of the code semantics while introducing sufficient disturbances; for context adaptability, consider the specific usage environment of the variable in the code to avoid generating semantically contradictory replacements in a specific context.

[0064] In some preferred embodiments, the following method for constructing a second set of replacement variables using a multi-layer screening strategy is provided, including:

[0065] S301. From the first set of replacement variables, extract variables whose lengths differ from the original variables whose importance scores are greater than the first preset threshold but are within a second preset threshold, thereby constructing a third set of replacement variables. This step ensures that the replacement variables are similar in length to the original variables, avoiding changes in code structure caused by excessive differences in variable lengths. The second preset threshold can be set to a reasonable range, such as ±2 or ±3 characters, to ensure that the replaced code remains essentially consistent in visual form, thereby more accurately assessing the model's sensitivity to semantic changes in variable names.

[0066] In another preferred embodiment, the method of the present invention adopts an adaptive calculation mechanism to determine the first preset threshold, that is, the threshold is dynamically adjusted according to the characteristics of different code samples and the model prediction confidence. Specifically, the Bootstrap sampling technology can be combined to calculate the variable importance distribution through multiple random samplings, and then the threshold is set based on the distribution characteristics. For samples with high prediction confidence, the threshold can be appropriately raised to focus on the most critical variables; for samples with low prediction confidence, the threshold can be lowered to consider more potential influencing factors. This dynamic threshold mechanism significantly improves the accuracy of variable screening, making the evaluation process more adaptable to code samples of different complexities.

[0067] S302. Obtain an embedding vector representation of each variable in the third replacement variable set, and apply the K-means clustering algorithm to cluster semantically similar variables in the third replacement variable set into K candidate clusters. This step utilizes modern natural language processing techniques to convert variable names into points in a high-dimensional vector space, and then cluster and group them based on semantic similarity. Variable embedding vectors can be obtained using a pre-trained word embedding model or an embedding model specifically designed for code, so that semantically similar variables are mapped to similar locations in the vector space. In a further optimization of this preferred embodiment, when applying the K-means clustering algorithm to cluster the third replacement variable set, Euclidean distance is used as the similarity metric between the embedding vectors of each variable. Euclidean distance, as a classic distance metric, can effectively measure the straight-line distance between points in a vector space and is suitable for quantifying the degree of semantic similarity between variable embedding vectors. Compared to other metrics such as cosine similarity, this metric more intuitively expresses the semantic proximity between variables, facilitating the selection of appropriate replacement variables in subsequent steps.

[0068] S303. Select at least one variable from each candidate cluster to form a second set of replacement variables. This step ensures that the final set of replacement variables has sufficient semantic diversity, as variables from different clusters have significant semantic differences. By selecting representative variables from each cluster, the present method constructs a comprehensive replacement set that includes both semantically similar and semantically distinct variables, providing a rich selection space for subsequent variable replacement.

[0069] S4. Iteratively perform variable replacement on the source code sample using the second set of replacement variables, generate and evaluate test samples in each iteration, and continuously optimize the test samples based on the difference in prediction results between the test samples in each round and the source code samples of the code model to be evaluated until the robustness evaluation is completed.

[0070] Step S4 is the core execution phase of the method. It systematically evaluates the robustness of the code model by generating a test sample set through iterative variable substitution. Based on the previously constructed second set of substitution variables, the code sample space is searched for adversarial examples that could cause the model's predictions to change, thereby comprehensively assessing the model's sensitivity to variable naming changes.

[0071] In some preferred embodiments, the steps include:

[0072] S401. Initialize a test sample set and add the source code sample to the test sample set. The original source code sample is added to the test sample set as the initial test sample, serving as the basis for subsequent variable replacement operations. The initial test sample inherits all the grammatical structure, semantic information, and execution logic of the original source code, ensuring that subsequently generated variants differ only in variable naming while maintaining complete functional logic.

[0073] S402. Each sample in the test sample set is traversed. For the initial variable currently being processed, the candidate replacement variable from the second replacement variable set is used to replace the initial variable. A new test sample is generated and added to the temporary sample set. In this step, the initial variable to be processed can be selected using various strategies. In some preferred embodiments, initial variables with importance scores greater than a first preset threshold θ are extracted and sorted in descending order of importance. These variables are then selected in this order in each iteration as the initial variables currently being processed. This strategy prioritizes variables that have the most significant impact on model predictions, effectively improving evaluation efficiency.

[0074] S403. Use the code model to be evaluated to test each sample in the temporary sample set, and calculate the difference between the predicted probability distribution of each sample and the source code sample. This difference between the predicted probability distribution of each sample and the source code sample refers to the statistical difference between the prediction results of the code model to be evaluated for the test sample after variable substitution and the original source code sample in the present invention method. Specifically, this difference quantifies the impact of variable substitution on the model output distribution and is a core indicator for evaluating the code model's sensitivity to variable naming. When the code model to be evaluated processes the source code sample and the test sample, it outputs the probability distribution of each category label. In some preferred embodiments, the difference value calculation method has two cases: when the predicted label of the test sample is the same as the predicted label of the source code sample, the difference value is calculated as the difference in the original label probabilities, that is, I(var_i) = P(y_true|X) - P(y_true|X'); when the predicted label of the test sample changes, the difference value calculation is more complex, and it is necessary to consider both the decrease in the original label probability and the increase in the new label probability, that is, I(var_i) = [P(y_true|X) - P(y_true|X')] - [P(y_other|X') - P(y_other|X)]. This difference value calculation method not only considers the change in probability value but also reflects whether the change is sufficient to cause a change in the prediction result, thereby comprehensively evaluating the decision boundary stability of the model. During the improved beam search process, samples with larger difference values ​​are considered more likely to reveal model robustness weaknesses and are therefore preferentially retained as the basis for the next round of iterations, effectively discovering areas where the model is sensitive to variable naming changes.

[0075] S404. Merge the test sample set retained from the previous iteration with the currently generated temporary sample set, and select the top B samples with the largest difference values ​​as the new test sample set. This step implements the dynamic evolution of the test sample set. The test sample set retained from the previous iteration is merged with the currently generated temporary sample set, and the top B samples with the largest difference values ​​are selected as the new test sample set. Parameter B is the number of samples retained and is a key hyperparameter that controls the breadth of the search. Larger B values ​​facilitate comprehensive exploration of the variable replacement space, while smaller B values ​​provide a more focused search and accelerate the discovery of adversarial samples.

[0076] In traditional beam search algorithms, each iteration only considers samples generated in the current round. However, the improved method of the present invention merges two sample sets during the sample selection phase of each iteration: the B most promising test samples retained after evaluation in the previous iteration (the test sample set), and a series of new samples generated in the current round by performing variable substitution on these B samples (the temporary sample set). This merging operation significantly expands the size and diversity of the candidate sample pool, thereby creating a larger search space. In some preferred embodiments, when the beamwidth parameter B is set to a large value (e.g., B=10), the merged candidate pool may contain hundreds of candidate samples. The system then selects the top B samples with the largest difference values ​​from this expanded candidate pool as the basis for the next iteration. This innovative merging mechanism effectively addresses the problem of traditional beam search and greedy algorithms easily falling into local optima by allowing temporarily ignored search paths to be "revived" in subsequent iterations. When some previously discarded samples, combined with new variable substitution, potentially produce more desirable evaluation results, they have the opportunity to be reconsidered. This dynamic integration strategy not only maintains the breadth and diversity of the search, but also controls the computational complexity through the beam width parameter, providing a more comprehensive and efficient technical solution for code model robustness evaluation.

[0077] S405. Repeat steps S402 to S404 until a stopping condition is met. The stopping condition refers to the criteria used by the improved beam search algorithm to terminate the current search process during iterative variable substitution testing. Specifically, the stopping condition is a set of preset judgment rules used to determine when to end the iterative evaluation of the code model robustness. During the improved beam search process, the system continuously replaces important variables and evaluates the model response. The stopping condition determines when this process should be terminated, ensuring the adequacy of the evaluation while avoiding unnecessary waste of computing resources. In some preferred embodiments, the stopping condition includes three conditions: First, when a test sample is found that can change the predicted results of the code model to be evaluated, it indicates that the model's robustness weaknesses have been successfully identified, and the search can be stopped immediately. Second, when all initial variables with importance scores greater than a first preset threshold have completed substitution testing, it indicates that all possible substitutions for important variables have been exhausted, and further testing is meaningless. Finally, when the number of iterations reaches a preset upper limit, the system will terminate the search process to avoid excessive consumption of computing resources. The setting of these three stopping conditions ensures that the evaluation process can not only comprehensively detect the robustness of the model, but also be completed efficiently within a reasonable computational complexity.

[0078] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.< / mask>

Claims

1. A code model robustness evaluation method based on variable importance analysis, characterized in that: Including steps: S1. Obtain the code model and source code sample to be evaluated; extract variables from the real source code library to construct a first set of replacement variables; S2. Calculate the importance score of each initial variable in the source code sample to the output of the code model to be evaluated, wherein the importance score is calculated based on the soft label method; S3. Filtering the first set of replacement variables for the initial variables whose importance scores are greater than a first preset threshold value to construct a second set of replacement variables; S4. Iteratively performing variable substitution on the source code sample using the second set of substitution variables, generating and evaluating test samples in each iteration, and continuously optimizing the test samples based on the difference in prediction results between the test samples and the source code samples in each iteration of the code model to be evaluated, until the robustness evaluation is completed; Step S4 includes: S401. Initialize the test sample set and add the source code sample to the test sample set; S402. Traverse each sample in the test sample set, replace the initial variable currently being processed with the candidate replacement variable in the second replacement variable set, generate a new test sample and add it to the temporary sample set; S403. Use the code model to be evaluated to test each sample in the temporary sample set, and calculate the difference value of the predicted probability distribution of each sample and the source code sample; S404. Combine the test sample set retained in the previous iteration with the currently generated temporary sample set, and select the top B samples with the largest difference values ​​as the new test sample set; S405. Repeat steps S402 to S404 until the stop condition is met.

2. The code model robustness evaluation method based on variable importance analysis according to claim 1, characterized in that: The method for constructing the second set of substitution variables includes: S301. From the first set of replacement variables, extract variables whose lengths differ from the initial variable lengths whose importance scores are greater than the first preset threshold and are within a second preset threshold range, and construct a third set of replacement variables; S302. Obtain an embedded vector representation of each variable in the third replacement variable set, and apply a K-means clustering algorithm to cluster semantically similar variables in the third replacement variable set into K candidate clusters; S303 . Select at least one variable in each candidate cluster to form a second replacement variable set.

3. The code model robustness evaluation method based on variable importance analysis according to claim 1, characterized in that: The method for selecting the initial variables of the current process in step S402 includes: Initial variables with importance scores greater than a first preset threshold are extracted and sorted in descending order of importance scores. In each iteration, they are selected in sequence as initial variables for current processing in the sorted order.

4. The code model robustness evaluation method based on variable importance analysis according to claim 1, characterized in that: The stop conditions include: Discover test samples that can change the predictions of the code model being evaluated; All initial variables with importance scores greater than the first preset threshold have been processed; The number of iterations reaches a third preset threshold.

5. The code model robustness evaluation method based on variable importance analysis according to claim 1, characterized in that: The method for calculating the importance score of each initial variable in the source code sample to the output result of the code model to be evaluated in step S2 includes: replacing the initial variables to be calculated in all instances of the source code sample with special tags to ensure the compilability of the source code sample and comprehensively calculate the importance of the initial variables to be calculated.

6. The method for evaluating the robustness of a code model based on variable importance analysis according to claim 2, wherein: When the K-means clustering algorithm is applied to cluster the third set of substitution variables in step S302, the Euclidean distance is used as the similarity metric between the embedding vectors of each variable.

7. A code model robustness evaluation system based on variable importance analysis, characterized by: The system is used to implement the code model robustness evaluation method based on variable importance analysis according to any one of claims 1 to 6.

Images