An attack method for a large language model vulnerability auditing tool based on interference attention

Abstract

The application discloses an attack method for a large language model vulnerability auditing tool based on interference attention, which comprises the following steps: performing function-level attention calculation on a sample code dataset to generate a high-attention function code dataset, selecting a highest-attention function from each function-level attention source code file of the high-attention function code dataset, and completing a compilation dependency to form a high-attention code segment. The function-level attention calculation on the sample code dataset improves the attention calculation efficiency by means of a dimension reduction algorithm; the high-attention function is selected and the compilation dependency is completed, so that the high-attention function can be successfully compiled and the concealment is improved; the method has obvious cross-model applicability and multi-programming language compatibility, can effectively reduce the accuracy of the large language model in the code vulnerability detection task, and has the advantage of improving the vulnerability code concealment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer security, in particular to an attack method for a vulnerability auditing tool based on a large language model based on interference attention. BACKGROUND

[0002] With the progress of the related capabilities of large language models in the field of computer security, large language models are increasingly being applied to vulnerability detection tasks. After being trained using a dataset composed of a large amount of security code and vulnerability code, large language models can detect potential vulnerabilities in code across scenarios. Today, large language models are embedded in integrated development environments to provide security recommendations and risk prompts for code written by developers in real time. For example, GitHub Copilot is a code assistance tool based on a large language model jointly developed by GitHub, Microsoft and OpenAI. It supports code vulnerability detection and can be embedded as a plug-in in the integrated development environment VS Code. As of March 2024, GitHub Copilot has more than 13 million paying users and more than 50,000 enterprise-level users.

[0003] However, vulnerability auditing tools based on large language models introduce new attack surfaces. Consider the following scenario: a developer first downloads code from an online community (such as GitHub) to a local computer, then uses a vulnerability auditing tool based on a large language model to detect the security of the code, and after checking, uses the code for their own development project (such as a web server). If a piece of vulnerability code can bypass the detection of a vulnerability auditing tool based on a large language model, the vulnerability can be hidden in the developer's project. Then in the future, an attacker can exploit the vulnerability to attack the developer's project.

[0004] The challenge of attacking vulnerability auditing tools based on large language models lies in the fact that large language models learn a wide variety of vulnerability codes during the training phase and have strong reasoning capabilities. Therefore, as long as the principles of the vulnerability are similar, vulnerabilities that have not been seen during the training phase can be detected. Therefore, attack methods (such as modifying variable names, code obfuscation) against traditional machine learning and deep learning vulnerability detection tools are difficult to work.

[0005] In summary, attacking vulnerability auditing tools based on large language models is challenging. Once the attack is successful, it is highly covert and destructive, and has practicality in the field of computer security. SUMMARY

[0006] The present application aims to provide an attack method for a vulnerability auditing tool based on a large language model based on interference attention, to solve the technical problems raised in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] S1. Perform function-level attention computation on the sample code dataset to generate a high-attention function code dataset. The sample code dataset contains several sample source code files C; specifically, each sample source code file C contains several functions. and several lines The high attention function code dataset contains several function-level attention source code files C', where each function-level attention source code file C' contains functions. The corresponding function-level attention;

[0009] S2. Select the highest attention function from each function-level attention source file C' in the high attention function code dataset and complete the compilation dependencies to form a high attention code snippet;

[0010] S3. After completing the compilation dependencies, the high-attention code snippet is integrated with the vulnerability code dataset and submitted to a vulnerability auditing tool based on a large language model for auditing and evaluation of the attack success rate.

[0011] Furthermore, the specific steps of step S1 are as follows:

[0012] S11, Original Attention Generation: The original attention tensor of the large language model for the sample source code file C is... Where T represents the number of tokens of C after the large language model segments C, L represents the number of Transformer layers of the large model, and H represents the number of attention heads of the large model;

[0013] S12, Hierarchical Attention Generation: First, the original attention... Summing along the fourth dimension yields the multi-head integrated attention. Then, use Using the last token in the first dimension (T tokens) as the query key and the T tokens in the second dimension as the query keys, we select the two-dimensional tensor corresponding to the last token in the first dimension (T tokens) to obtain the hierarchical attention of all tokens. ;

[0014] S13, Row-level Attention Generation: Generating hierarchical attention The first dimension of the T tokens is indexed according to the row they belong to. Summation yields a one-dimensional row-level attention. Then, perform one-dimensional row-level attention. Summing the first dimension and merging all attention layers yields the row-level attention for each row. where i∈[1,M];

[0015] S14, function-level attention generation: summing up the row-level attention of each function to obtain the function-level attention of each function according to the function to which each row belongs summing up to obtain the function-level attention of each function wherein j∈[1, N];

[0016] S15, performing S11-S14 on each sample source code file C in the sample code data set to obtain a function-level attention source code file C', forming a high-attention function code data set.

[0017] Furthermore, the highest attention function in each function-level attention source code file C' in the high-attention function code data set is a candidate, thereby generating a batch of high-attention function candidates; the compilation dependencies of the high-attention function are completed using a large language model, and the high-attention function and its compilation dependencies after completion can be self-compiled.

[0018] Furthermore, the step S3 of evaluating the attack success rate is specifically:

[0019] S31, marking the answers of the vulnerability audit tool based on the large language model when auditing a piece of vulnerability code with scores; specifically, marking 1 point for not finding any vulnerabilities; marking 2 points for finding several vulnerabilities, but all of them are false positives and do not contain correct vulnerabilities; marking 3 points for finding several vulnerabilities, and there are no other false positives, and the found vulnerabilities are correct vulnerabilities; marking 4 points for finding several vulnerabilities, and there are false positives, but also contain correct vulnerabilities;

[0020] S32, according to whether the vulnerability exists and whether the discovered vulnerability is correct, dividing the answers and corresponding scores into two levels of indicators to evaluate whether the vulnerability audit tool based on the large language model is successful, wherein the indicators include vulnerability existence level and type correctness level;

[0021] S33, defining two success rates BSR@exist and BSR@type to evaluate the attack success rate of the high-attention code segment on the vulnerability audit tool based on the large language model.

[0022] Furthermore, the step S32 of evaluating whether the vulnerability audit tool based on the large language model is successful is specifically:

[0023] In the vulnerability existence level: scores of 2, 3 or 4 are successful in discovering vulnerabilities, and the audit is successful; a score of 1 is not successful in discovering vulnerabilities, and the audit fails;

[0024] In the type correctness level: scores of 3 or 4 are successful in discovering correct vulnerabilities, and the audit is successful; scores of 1 or 2 are not successful in discovering correct vulnerabilities, and the audit fails.

[0025] Further, the success rate evaluation in the step S33 is specifically:

[0026] BSR@exist evaluation: at the vulnerability existence level, the number of cases that fail the audit after integrating the high-attention code fragments is divided by the number of cases that succeed the audit before integrating the high-attention code fragments;

[0027] BSR@type evaluation: at the type correctness level, the number of cases that fail the audit after integrating the high-attention code fragments is divided by the number of cases that succeed the audit before integrating the high-attention code fragments.

[0028] Beneficial effects: the present application performs function-level attention calculation on the sample code data set, improves the attention calculation efficiency by means of the dimension reduction algorithm, selects high-attention functions and completes the compilation dependencies, so that the high-attention functions can be successfully compiled and the concealment is improved, the method has significant cross-model applicability and multi-programming language compatibility, can effectively reduce the accuracy of large language models in the code vulnerability detection task, and has the advantages of improving the vulnerability of the code. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions implemented by the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0030] Figure 1 Flowchart of the attack method based on interference attention of the present application for the attack method based on the large language model vulnerability audit tool;

[0031] Figure 2 Table of the sample code data set and the vulnerability code data set of the present application;

[0032] Figure 3 Large language model details table of the present application;

[0033] Figure 4 Attack effect table of the attack method and the benchmark method of the present application on 5 open source large language models on the Smart-bugs vulnerability data set;

[0034] Figure 5 Result table of the attack method of the present application verifying the cross-model migration;

[0035] Figure 6 Result table of the attack method of the present application verifying the cross-language migration;

[0036] Figure 7 The semantic similarity comparison result table of the vulnerability code before and after the integration of the high attention code segment of the application;

[0037] Figure 8 The attention drop amount and drop rate result table of the vulnerability code before and after the integration of the high attention code segment of the application with the highest attack success rate. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the application will be apparently and completely described in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0039] As shown in Figures 1-8 The application provides an attack method for a vulnerability audit tool based on a large language model based on interference attention, and the specific steps are as follows:

[0040] Please refer to Figure 2 In this embodiment, a sample code dataset for generating a high attention code segment is downloaded, including an intelligent contract dataset Messiq, a C / C++ dataset Leetcode-cpp and a Python dataset Leetcode-py.

[0041] A vulnerability code dataset for attacking developers is downloaded, including an intelligent contract vulnerability dataset Smart-bugs, a C / C++ vulnerability dataset Big-Vul and a Python vulnerability dataset CVE-Fixes.

[0042] Since the large model has a limit on the number of tokens when calculating attention, the source code files with a total number of tokens less than 4096 are screened.

[0043] Please refer to Figure 3 Pretrained large language models are downloaded from the large language model open source community Huggingface, and local deployment is completed. There are 5 open source large language models (Mistral, MixtralExpert, Gemma, CodeLlama and Phi). The access permission API Key of GPT-4o is purchased, so that the closed source large language model GPT-4o can be used through the API Key.

[0044] S1, function-level attention calculation is performed on the sample code dataset to generate a high attention function code dataset, wherein the sample code dataset contains a plurality of sample source code files C; specifically, each sample source code file C contains a plurality of functions and several rows , the high attention function code data set contains several function-level attention source code files C', wherein each function-level attention source code file C' contains a function corresponding function-level attention, each sample source code file C can contain a vulnerability or not, and is only used to generate high attention functions;

[0045] In this embodiment, in order to describe the implementation of step S1, the specific steps are as follows:

[0046] S11, original attention generation: the original attention tensor of the sample source code file C by the large language model is , wherein T represents the token number of C after the large language model tokenizes C, L represents the number of Transformer layers of the large model, H represents the number of attention multi-heads of the large model, T in the first dimension of A as the query key, and T in the second dimension as the queried core key;

[0047] S12, hierarchical attention generation: first, the original attention is summed up according to the fourth dimension to obtain the multi-head integrated attention , which can be expressed by the formula as follows:

[0048] ;

[0049] Then, using the last token in the first dimension T tokens as the query key and the T tokens in the second dimension as the queried key, the two-dimensional tensor corresponding to the last token in the first dimension T tokens is selected to obtain the hierarchical attention of all tokens , which can be expressed by the formula as follows:

[0050] ;

[0051] S13, row-level attention generation: the first dimension T tokens in the hierarchical attention are summed up according to the index of the row where the token is located to obtain one-dimensional row-level attention ; then, the first dimension of the one-dimensional row-level attention is summed up to combine all attention layers to obtain the row-level attention of each row , wherein i∈[1, M], which can be expressed by the formula as follows:

[0052] ,

[0053] S14, function-level attention generation: the row-level attention is summed up according to the function to which each row belongs Summing up, the function-level attention of each function is obtained wherein, j∈[1, N], the formula can be expressed as:

[0054] S15, performing S11-S14 on each sample source code file C in the sample code data set to obtain a function-level attention source code file C', and forming a high-attention function code data set.

[0055] S2, selecting the highest attention function from each function-level attention source code file C' in the high-attention function code data set and completing the compilation dependency to form a high-attention code snippet; specifically, the highest attention function in each function-level attention source code file C' is a candidate, thereby generating a batch of high-attention function candidates; using a large language model (such as GPT-4o) to complete the compilation dependency of the high-attention function, and the high-attention function and its compilation dependency after completion can be self-compiled.

[0056] S3, integrating the high-attention code snippet after completing the compilation dependency with the vulnerability code data set and submitting it to a vulnerability audit tool based on a large language model for auditing, evaluating the attack success rate, and the standard for attack success is that the vulnerability audit tool based on a large language model cannot detect the existence of vulnerabilities in the vulnerability code; in order to make the vulnerability audit tool based on a large language model in the real world, some methods for improving the quality of answers will be used in the design of the prompt words, such as Retrieval-Augmented-Generation, Chain-of-Thought, and In-Context-Learning.

[0057] The embodiment designs an evaluation attack success rate for evaluating the vulnerability audit tool based on a large language model, as follows:

[0058] S31, marking the answers of the vulnerability audit tool based on a large language model when auditing a piece of vulnerability code with scores, and the embodiment designs four answers, which are marked with scores 1-4 in turn, specifically: no vulnerabilities are found, marked as 1; several vulnerabilities are found, but all are false positives and do not contain correct vulnerabilities, marked as 2; several vulnerabilities are found, and there are no other false positives, and the vulnerabilities found are correct vulnerabilities, marked as 3; several vulnerabilities are found, and there are false positives, but also contain correct vulnerabilities, marked as 4.

[0059] S32, according to whether the vulnerability exists and whether the discovered vulnerability is correct, the answers and corresponding scores are divided into two levels of indicators to evaluate the audit success of the vulnerability audit tool based on a large language model, wherein the indicators include vulnerability existence level and type correctness level. ​

[0060] The embodiment evaluates whether the vulnerability audit tool based on the large language model is successful in auditing, specifically:

[0061] At the vulnerability existence level: a score of 2, 3 or 4 is a successful discovery of a vulnerability, and the audit is successful; a score of 1 is not a successful discovery of a vulnerability, and the audit fails, and this level measures the detection ability of the large language model on the existence of a vulnerability;

[0062] At the type correctness level: a score of 3 or 4 is a successful discovery of a correct vulnerability, and the audit is successful; a score of 1 or 2 is not a successful discovery of a correct vulnerability, and the audit fails, and this level measures the detection ability of the large language model on the type of a correct vulnerability.

[0063] S33, define two attack success rates BSR@exist and BSR@type to evaluate the attack success rate of the high attention code segment on the vulnerability audit tool based on the large language model. Among them, BSR@exist is used to evaluate: at the vulnerability existence level, the number of cases in which the vulnerability dataset fails to audit (score 1) after integrating the high attention code segment divided by the number of cases in which the audit is successful (score 2, 3 or 4) before integrating the high attention code segment; BSR@type is used to evaluate: at the type correctness level, the number of cases in which the vulnerability dataset fails to audit (score 1 or 2) after integrating the high attention code segment divided by the number of cases in which the audit is successful (score 3 or 4) before integrating the high attention code segment.

[0064] Verify the effectiveness of the attack method proposed in the application, specifically as follows:

[0065] Using the attack method proposed in the application and two baseline methods (random insertion (inserting irrelevant interference code in the vulnerability function) and renaming (harmless renaming of variables in the vulnerability code)), attack different large language models on the Smart-bugs vulnerability dataset, and evaluate the effectiveness of the attack according to the attack success rates defined by BSR@exist and BSR@type.

[0066] As Figure 4As shown, this embodiment demonstrates the performance comparison of the attack method proposed in this invention on different large language model vulnerability auditing tools. Each column reflects the success rate of this invention compared to the baseline method on the Smart-bugs dataset. Regardless of whether BSR@exist or BSR@type is used, this invention outperforms other baseline methods on all five models, demonstrating consistent efficiency. Specifically: using BSR@exist as the evaluation metric, the attack success rate of this invention on the Phi model increased from 0% for the random insertion method to 17.32%. Compared to the renaming method, the attack success rate of this invention on the Mistral model increased from 0% to 45.31%, and the improvement ranged from 1.36 times (CodeLlama) to 49.14 times (Gemma) on other models. When using BSR@type as the evaluation metric, this invention consistently outperforms the random method, with an improvement range from 1.77 times (CodeLlama) to 43.99 times (Phi). Compared to renaming methods, the success rate of attacks using this invention is improved by 1.27 times (CodeLlama) to a maximum of 73 times (Mistral).

[0067] The cross-model transferability of the attack method proposed in this invention is verified as follows:

[0068] The transferability of the high-attention code snippets that performed best on a single model to other models was evaluated. Specifically, this embodiment selected the high-attention code snippets with the highest BSR@exist and BSR@type, respectively. These snippets were then tested for attack effectiveness on other models using the Smart-bugs vulnerability dataset as the evaluation benchmark. The results show that the present invention has strong transferability across different models.

[0069] like Figure 5 As shown, this embodiment demonstrates the attack effectiveness of the optimal high-attention code snippet on each model on other models. Under the BSR@exist metric, the optimal high-attention code snippet on the Gemma model exhibits the highest transferability, achieving an attack success rate of 87.5% on the Mistral model and 18.11% on the Phi model. Similarly, under the BSR@type metric, the optimal high-attention code snippet on the Gemma model also maintains a high success rate, exceeding 90% on the Mistral model and exceeding 40% on the Phi model. Notably, the optimal high-attention code snippet on the Gemma model also shows a 30% attack success rate against the closed-source model GPT-4o. These results demonstrate that the attack method generated in this invention possesses strong cross-model transferability, exhibiting high effectiveness against both open-source and closed-source models.

[0070] To verify the cross-language transferability of the attack method proposed by the present application, the following is done:

[0071] To verify the scalability of the attack method proposed by the present application on different languages, this embodiment plans to test the effectiveness of the attack method proposed by the present application on additional datasets containing other languages (such as C, C++, and Python) using the same experimental setup as in verifying the effectiveness of the attack method proposed by the present application. The vulnerability audit tools based on large language models used for scalability evaluation include Mixtral, CodeLlama, and Phi. Similarly, BSR@exist and BSR@type are selected as evaluation indicators.

[0072] As shown in Figure 6 , the present application performs well in terms of attack success rate, especially on the Big-Vul dataset, reaching 100% in both indicators (BSR@exist and BSR@type) of the Phi model. In contrast, the random insertion and renaming methods have lower overall performance on the Phi model, with success rates below 50% in multiple cases.

[0073] To verify the stealthiness of the attack method proposed by the present application, the following is done:

[0074] To evaluate the stealthiness of the attack method proposed by the present application, this embodiment measures the semantic similarity before and after integrating high-attention code snippets on multiple vulnerability code datasets. The best-performing high-attention code snippets on each target open-source model are used, and are inserted into code from different datasets (including Smart-bugs (Solidity), Big-Vul (C / C++), and CVE-Fixes (Python)). To quantify semantic changes, a frozen language model (FLM) is used, specifically Alibaba-NLP / gte-large-en-v1.5, to evaluate semantic similarity by calculating the cosine similarity of embedding vectors.

[0075] As shown in Figure 7 , this embodiment shows the semantic similarity values (minimum, maximum, and mean) of each model on different datasets. Overall, the present application exhibits high stealthiness, with high mean similarity scores on each dataset indicating that the stealthiness of code modifications is high enough that developers cannot easily detect it. For example, CodeLlama consistently achieves the highest mean similarity in each dataset, with a value of approximately 0.92 or higher, indicating minimal semantic deviation after insertion. In contrast, the similarity score of Phi is slightly lower, especially on the Smart-bugs dataset (with an average of 0.8179), but still maintains a high level of similarity. These results indicate that the present application can effectively preserve the semantics of the original code, thereby achieving a high level of stealthiness.

[0076] The rationality of the attack method proposed in this invention is verified as follows:

[0077] To address the question of whether a successfully attacked high-attention code snippet actually leads to a decrease in function-level attention of the vulnerable function, this embodiment validates the best high-attention code snippet on each model on the Smart-bugs dataset, verifying the amount and rate of decrease in function-level attention of the vulnerable function before and after its integration into the vulnerable code.

[0078] The impact of high-attention code snippets on the attention of vulnerable code was verified by: calculating the function-level attention of the vulnerable function in the vulnerable code before and after integrating the high-attention code snippets, thus verifying the effectiveness of the high-attention code snippets in diverting attention; calculating the attention of the vulnerable function: calculating the function-level attention of the vulnerable function before and after integrating the high-attention code snippets; and predicting the effectiveness of the attack: predicting the effectiveness of the attack by calculating the decrease and rate of decrease in the function-level attention of the vulnerable function.

[0079] like Figure 8 As shown, in the Phi model, the high-attention code snippets resulted in the largest decrease in function-level attention of the vulnerable function, reaching 325.89; while in the Gemma model, the high-attention code snippets achieved the largest decrease rate, at 0.92. This demonstrates that the high-attention code snippets generated by this invention can significantly reduce the function-level attention of the vulnerable function, thereby helping it escape vulnerability auditing by large language models.

[0080] This invention discloses an attack method for large language model-based vulnerability auditing tools based on interference attention. It performs function-level attention calculations on a sample code dataset, using dimensionality reduction algorithms to improve the efficiency of attention calculations. High-attention functions are selected and their compilation dependencies are supplemented, enabling successful compilation and improving stealth. This method has significant cross-model applicability and multi-programming language compatibility, effectively reducing the accuracy of large language models in code vulnerability detection tasks and improving the stealth of vulnerable code.

[0081] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0082] Furthermore, it should be understood that although the specification is described in terms of embodiments, not every embodiment includes only one independent technical solution, and the specification is described in this way only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. An attack method for a large language model-based vulnerability auditing tool based on interference attention, characterized in that, The method comprises the following steps: S1, performing function-level attention calculation on a sample code data set to generate a high-attention function code data set, wherein the sample code data set contains a plurality of sample source code files C; specifically, each sample source code file C contains a plurality of functions , and a plurality of lines , and the high-attention function code data set contains a plurality of function-level attention source code files C', wherein each function-level attention source code file C' contains a function corresponding function-level attention; The specific steps of the step S1 are: S11, original attention generation: the original attention tensor of the large language model to the sample source code file C is wherein, T represents the token quantity of C after the large language model performs tokenization on C, L represents the number of Transformer layers of the large model, and H represents the number of attention multi-heads of the large model. S12, hierarchical attention generation: first, the original attention Sum up along the fourth dimension to get multi-head integrated attention ; then, using The last token in the first dimension T tokens as the query key, the T tokens in the second dimension as the query key, select the two-dimensional tensor corresponding to the last token in the first dimension T tokens, get the hierarchical attention of all tokens ; S13, row-level attention generation: summing up the first dimension T tokens according to the index of the row where the token is located summing up to obtain one-dimensional row-level attention ; then, summing up the first dimension of the one-dimensional row-level attention to combine all attention layers and obtain the row-level attention of each row where i ∈ [1, M]​ S14, function-level attention generation: summing up the row-level attention of each function per function summing up to obtain the function-level attention of each function where j ∈ [1, N] S15, performing S11-S14 on each sample source code file C in the sample code data set to obtain a function-level attention source code file C', and forming a high-attention function code data set; S2, selecting the highest attention function from each function-level attention source code file C' in the high-attention function code data set and completing the compilation dependency to form a high-attention code snippet; S3, integrating the high-attention code snippet after the compilation dependency is completed with the vulnerability code data set, and submitting the integrated code to a vulnerability audit tool based on a large language model for auditing to evaluate the attack success rate.

2. The attack method of claim 1, wherein: The specific step S2 is that the highest attention function in each function-level attention source code file C' in the high-attention function code data set is used as a candidate, thereby generating a batch of high-attention function candidates; The compilation dependency of the high-attention function is completed using a large language model, and the high-attention function and the compilation dependency thereof are self-compiled after the completion.

3. The attack method of claim 1, wherein: The specific evaluation of the attack success rate in the step S3 is: S31, marking the answer of the vulnerability audit tool based on the large language model when auditing a piece of vulnerability code with a score; specifically, marking 1 point if no vulnerability is found, marking 2 points if a number of vulnerabilities are found but all are false positives and no correct vulnerability is found, marking 3 points if a number of vulnerabilities are found and no false positive is found and the found vulnerabilities are correct, and marking 4 points if a number of vulnerabilities are found and there are false positives but there are also correct vulnerabilities; S32, dividing the answer and the corresponding score into two levels of indexes to evaluate whether the vulnerability audit tool based on the large language model is successful, wherein the indexes include a vulnerability existence level and a type correctness level; S33, defining two success rates BSR@exist and BSR@type to evaluate the attack success rate of the high-attention code snippet on the vulnerability audit tool based on the large language model.

4. The attack method of claim 3, wherein: The specific evaluation of whether the vulnerability audit tool based on the large language model is successful in the step S32 is: In the vulnerability existence level, a score of 2, 3 or 4 indicates successful discovery of vulnerabilities and successful auditing, and a score of 1 indicates unsuccessful discovery of vulnerabilities and unsuccessful auditing; In the type correctness level, a score of 3 or 4 indicates successful discovery of correct vulnerabilities and successful auditing, and a score of 1 or 2 indicates unsuccessful discovery of correct vulnerabilities and unsuccessful auditing.

5. The attack method of claim 3, wherein: The specific evaluation using the success rate in the step S33 is: BSR@exist evaluation: in the vulnerability existence level, the number of cases in which the vulnerability data set fails in auditing after the high-attention code snippet is integrated is divided by the number of cases in which the vulnerability data set succeeds in auditing before the high-attention code snippet is integrated; BSR@type evaluation: in the type correctness level, the number of cases in which the vulnerability data set fails in auditing after the high-attention code snippet is integrated is divided by the number of cases in which the vulnerability data set succeeds in auditing before the high-attention code snippet is integrated.

Images