Static analysis rule automatic generation method based on LLM and RAG
By combining LLM and RAG, static analysis rules are automatically generated, solving the cost and accuracy problems of static analysis tools when detecting defects in specific business logic. This achieves efficient and accurate defect detection, reduces labor costs, and improves software quality.
Patent Information
- Application Number
- CN202511045096.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-11-11
AI Technical Summary
Existing static analysis tools struggle to detect defects related to specific business logic, and relying on manually defined rules is costly. LLM is computationally expensive in large-scale codebases and is difficult to effectively mine deep semantic information and combine it with contextual information.
By employing an LLM and RAG-based approach, rule description information is generated by acquiring static analysis rules and historical defect information. Semantic similarity matching is then performed, and static analysis tools are automatically selected to generate and adjust rules until they are effective, thereby reducing manual costs and improving detection accuracy.
It achieves accurate defect detection, reduces the manual cost of rule formulation, improves the comprehensiveness and accuracy of static analysis, is suitable for teams with limited computing resources, and improves software quality.
Smart Images
Figure CN120929352A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of software static analysis, and in particular to a method for automatically generating static analysis rules based on LLM and RAG. Background Technology
[0002] With the rapid development of information technology, the scale and complexity of software systems continue to rise, placing higher demands on software quality assurance. Static analysis, as an efficient code quality inspection method, can perform in-depth analysis of software before the testing phase, identifying potential defects in advance, thereby significantly improving code quality and shortening the verification cycle. However, the built-in rule sets of current mainstream static analysis tools are mainly designed for general software defects, while different software systems have significant differences in business logic and application scenarios, making it difficult for these tools to directly detect defects related to specific business logic. Therefore, in practical project applications, it is usually necessary to customize static analysis rules according to specific business needs to achieve more accurate defect detection.
[0003] Customizing static analysis rules typically requires not only business-related knowledge but also extensive knowledge of software analysis, making it difficult for developers or testers to handle. Furthermore, employing dedicated static analysts is quite costly, which significantly hinders the promotion and popularization of static analysis methods in business-related fields.
[0004] To address the difficulty of formulating rules for static analysis, various automatic rule extraction methods exist. These methods, based on program analysis or deep learning techniques, extract defect patterns from historical code fixes and then match these patterns against existing code. If a code segment with a high similarity to the defect pattern is found, it indicates a potential problem in the code. However, while these methods make relatively full use of statement dependencies and some semantic information, they struggle to uncover deeper semantic information and effectively eliminate noise during changes by incorporating contextual information. Furthermore, these static analysis rule generation methods often only generate rules for the same scanning tool. Therefore, manual selection of the scanning tool based on the type of software defect to be scanned, and consequently, the method for generating scanning rules, is required, relying heavily on the experience of the software defect analyst.
[0005] In recent years, the application of Large Language Models (LLMs) has become increasingly widespread, enhancing their ability to understand code and software documentation. Furthermore, large models possess deep semantic understanding capabilities, effectively identifying defects in code. However, most current methods use LLMs to understand source code or data generated by static analysis tools to determine the presence, type, and location of errors. However, these methods rely on the continuous scanning and reasoning of the code by the LLM, which often results in prohibitively high computational costs for large code repositories. Summary of the Invention
[0006] The purpose of this application is to provide an automatic generation method for static analysis rules based on LLM and RAG, which can effectively prevent the recurrence of historical defects, achieve accurate defect detection, and improve software quality.
[0007] To achieve the above objectives, this application provides the following solution:
[0008] This application provides a method for automatically generating static analysis rules based on LLM and RAG, including:
[0009] Acquire information data; the information data includes static analysis rules and corresponding descriptive information stored in the knowledge base, as well as historical software defect information;
[0010] LLM is used to generate rule description information based on the historical software defect information;
[0011] Based on LLM, the system performs summarization and optimization based on the rule description information, and matches static analysis tools based on semantic similarity to determine the static analysis tools.
[0012] Construct complete prompt terms; the complete prompt terms include: few sample case information, rule description information, type of static analysis tool, task description, and static analysis rule syntax specification information; the few sample case information is obtained by using a hybrid retrieval strategy that combines keyword similarity and semantic retrieval, performing keyword and vectorized semantic matching, sorting by similarity, and selecting a set number of static analysis rule contents and corresponding description information with high similarity values;
[0013] Based on the pre-trained large language model, the analysis rules of the static analysis tool are determined according to the complete prompt words, and the rules are scanned and verified using software historical defect information to obtain the verification results.
[0014] If the verification result indicates that the rule is invalid, the analysis rule is regenerated based on the pre-trained large language model. If the rule is invalid for a set number of times, the process returns to the step of "using LLM to generate rule description information based on the software's historical defect information". The prompt words for the LLM-generated rule description information are adjusted, and subsequent rule generation and scanning steps are executed until the verification result indicates that the rule is valid.
[0015] Based on the analysis rules of static analysis tools, the corresponding static analysis tools are invoked to scan the source code of the project to be analyzed and the defect reports generated by the tools are collected.
[0016] The results will be evaluated based on the defect report.
[0017] If the evaluation result is valid, the static analysis rules and corresponding descriptive information of the static analysis tool will be added to the knowledge base.
[0018] If the evaluation result is invalid, the analysis rules are first regenerated based on the pre-trained large language model. If there are invalid rules after a set number of attempts, the process returns to the step of "generating rule description information based on the software's historical defect information using LLM". The prompt words for generating the rule description information are then adjusted. After correcting the prompt word content, the subsequent steps are executed again until the evaluation result is valid.
[0019] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0020] This application provides an automatic generation method for static analysis rules based on LLM and RAG. The method includes: generating rule description information using LLM based on historical software defect information; retrieving similar rules based on semantic similarity as few-sample cases; combining these with the rule description information to construct complete prompt words; matching static analysis tools based on semantic similarity; determining the analysis rules of the static analysis tools based on the complete prompt words; and performing rule scanning and verification; based on the analysis rules of the static analysis tools, calling the corresponding static analysis tools to scan the source code of the project to be analyzed; evaluating the results based on the defect report; if valid, adding the static analysis rule content and corresponding description information of the static analysis tools to the knowledge base; if invalid, adjusting the prompt words in the rule description information, correcting the prompt word content, and re-executing the subsequent steps until the evaluation result is valid. This application can automatically select static analysis tools and generate corresponding analysis rules, reducing the manual cost of rule formulation. Furthermore, this application uses few-sample case information to provide prompts for a pre-trained large language model, thereby effectively combining the advantages of different static detection tools for software defect detection and improving the comprehensiveness and accuracy of static analysis. Therefore, this application enables accurate defect detection and improves software quality. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A flowchart illustrating the automatic generation method for static analysis rules based on LLM and RAG;
[0023] Figure 2 This diagram illustrates the technical steps of an automatic generation method for static analysis rules based on LLM and RAG. Detailed Implementation
[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] This application proposes an adaptive automatic generation method for static analysis rules based on LLM and RAG. Using a large model as the core, it analyzes historical defect repair information, automatically selects software defect scanning tools based on defect characteristics, and automatically generates static analysis rules based on the rule cases of the corresponding tools. The method described in this application can automatically match static detection methods corresponding to the static analysis functional requirements based on a knowledge base. Simultaneously, it uses similar rule cases retrieved as samples and employs a few-shot suggestion method to provide hints to the pre-trained large model, thus effectively combining the advantages of different static detection tools for software defect detection and improving the comprehensiveness and accuracy of static analysis. Furthermore, this application eliminates the need for further LLM intervention after the static analysis rules are formulated. Scanning based on static analysis rules primarily relies on general-purpose CPU and memory resources, making it more suitable for teams with limited computing resources. In summary, this application can automatically select static analysis tools and generate static analysis rules, reducing the manual cost of rule formulation. Compared with existing static analysis rule generation methods, it can more deeply mine the semantic features of programs. Compared with methods that directly analyze code based on LLM, it can effectively reduce the consumption of computing resources and provide software development teams with a novel, project-based direction for static analysis, thereby effectively assisting software quality control. Retrieval-Augmented Generation (RAG) is a technical framework that combines information retrieval and text generation, mainly used to improve the performance of Large Language Models (LLM) in knowledge-intensive tasks.
[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0027] In one exemplary embodiment, such as Figure 1 As shown, an automatic generation method for static analysis rules based on LLM and RAG is provided, including:
[0028] Step 100: Obtain information data. Information data includes static analysis rules and their corresponding descriptions stored in the knowledge base, as well as historical software defect information. Historical software defect information includes: correct / incorrect versions of the software source code, software defect descriptions and fixes; this information is stored in plain text format.
[0029] The knowledge base includes a static analysis rule knowledge base and a static analysis tool knowledge base. Both the static analysis rule knowledge base and the static analysis tool knowledge base use a hierarchical storage structure and contain a raw data layer, a metadata layer, and a vector embedding layer.
[0030] The raw data layer of the static analysis rule knowledge base contains predefined static analysis rules and corresponding functional descriptions. The metadata layer of the static analysis rule knowledge base is extracted from the raw data based on LLM and includes information on the code defect categories to which the rules apply, the applicable programming languages and development frameworks, the static analysis tools used, and the corresponding static analysis methods.
[0031] The raw data layer of the static analysis tool knowledge base contains descriptive information about the static analysis tools; the metadata layer of the static analysis tool knowledge base is extracted from the raw data based on LLM and contains information on the programming languages applicable to the static analysis tools, supported static analysis methods, and analysis types.
[0032] The vector embedding layers corresponding to the static analysis rule knowledge base and the static analysis tool knowledge base are obtained by embedding high-dimensional vectors into a large model after deduplication and standardization of the original data content.
[0033] Embedding high-dimensional vectors based on large models involves specific embedding methods, including:
[0034] The standardized original data text is converted into a word sequence using a word segmenter based on byte-pair encoding. After encoding using a Transformer layer, the hidden state of the last layer is selected as the word vector. The expression corresponding to each word vector is:
[0035] E out =E token +E position +E segment ;
[0036] Among them, E token E is the vector to which it is mapped. position For position encoding; E segment E serves as a marker for sentences. out Word vectors;
[0037] Based on the word vectors, the sentence vector is calculated using the mean pooling method; the sentence vector... H i Let be the word vector of the i-th word; T is the total number of words in the sentence.
[0038] Store the vector of each sentence into the corresponding vector database.
[0039] Step 200: Use LLM to generate rule description information based on historical software defect information.
[0040] The rule description information includes: functional rule description information and implementation details; the functional rule description information includes: the target programming language, the applicable programming framework and software module, the defect type, and the defect cause.
[0041] The implementation details include: the matching algorithm used, the matching mode, the types and semantic information of key variables, and the false positive cases that need to be excluded.
[0042] The rule description information is stored in plain text format in Markdown.
[0043] Step 300: Based on LLM, perform summarization and optimization processing according to the rule description information, and match static analysis tools based on semantic similarity to determine the static analysis tools.
[0044] Based on LLM, summarization and optimization are performed according to rule description information, and static analysis tools are matched based on semantic similarity to determine the static analysis tools, specifically including:
[0045] Based on LLM, the system performs summarization and optimization based on the rule description information to generate query keywords and query statements.
[0046] Based on the query keywords and query statements, static analysis tools are matched using semantic similarity to determine the static analysis tools. A hybrid retrieval strategy combining keyword similarity retrieval and semantic retrieval is employed to search the knowledge base of static analysis tools and match the static analysis tools. The hybrid retrieval strategy includes: performing query keyword retrieval and filtering, as well as performing vectorized retrieval.
[0047] Vectorized retrieval includes: using the cosine similarity between the query vector u and the vector v in the database. Perform a match.
[0048] The data is sorted from highest to lowest similarity, and the matching data corresponding to the highest similarity is selected as the static analysis tool.
[0049] Step 400: Construct complete suggestion terms. That is, construct complete suggestion terms based on RAG. Complete suggestion terms include: few-sample case information, rule description information, type of static analysis tool, task description, and static analysis rule syntax specification information; few-sample case information is obtained by using a hybrid retrieval strategy combining keyword similarity and semantic retrieval. After keyword and vectorized semantic matching, the results are sorted by similarity, and a set number of static analysis rules with high similarity scores and their corresponding descriptions are selected.
[0050] Step 500: Based on the pre-trained large language model, determine the analysis rules of the static analysis tool according to the complete prompt words, and use the software's historical defect information to scan and verify the rules, and obtain the verification results.
[0051] Step 600: If the verification result is that the rule is invalid, the analysis rule is regenerated based on the pre-trained large language model. If there are invalid rules after a set number of times, the process returns to the "Generate rule description information based on software history defect information using LLM" step, adjusts the prompt words for the LLM-generated rule description information, and executes subsequent rule generation and scanning steps until the verification result is that the rule is valid.
[0052] The conditions for determining whether the validation result is valid for the rule include:
[0053] Whether the scan can be completed normally without exiting abnormally due to rule syntax errors or tool runtime errors; whether the corresponding type of defect is found in the defective version code; and whether the corresponding defect is not reported in the version after the defect is fixed.
[0054] Step 700: Based on the analysis rules of the static analysis tool, invoke the corresponding static analysis tool to scan the source code of the project to be analyzed and collect the defect reports generated by the tool. The analysis rules of the static analysis tool are configured in the form of configuration files, including YAML and XML. In addition, as an optional implementation, the analysis rules of the static analysis tool are in binary form, including executable files and dynamic link libraries.
[0055] Step 800: Evaluate the results based on the defect report.
[0056] Step 900: If the evaluation result is valid, the static analysis rules and corresponding description information of the static analysis tool are added to the knowledge base.
[0057] Step 1000: If the evaluation result is invalid, the analysis rules are regenerated based on the pre-trained large language model. If there are invalid rules after a set number of attempts, the process returns to the step "Generate rule description information using LLM based on the software's historical defect information". The prompt words for generating the rule description information are adjusted. After correcting the prompt word content, the subsequent steps are executed again until the evaluation result is valid.
[0058] The criteria for determining the validity of the evaluation results include: generating a valid code defect report after actual verification and analysis; no issues were found with the rules; and the number of false positives is less than a preset threshold TH. MaxFP .
[0059] This application proposes an adaptive automatic generation method for static analysis rules based on LLM and RAG. It analyzes historical defect repair information using a large model and automatically generates static analysis rules. For example... Figure 2 As shown, the specific operating steps in practical applications are as follows.
[0060] Step 1: Collect the pre-defined (expert-defined) static analysis rules and their descriptions to initialize the static analysis rule knowledge base. The knowledge base contains the functional descriptions, detailed explanations, and content of the static analysis rules. If a rule lacks a functional or detailed description, LLM is used to reverse-engineer the corresponding description based on the rule. After manual review and adjustment, the description is stored in the knowledge base. Simultaneously, collect historical software defect information, including correct / incorrect versions of the software source code, software defect descriptions, and repair information, and store this information in plain text format.
[0061] Step 1, "Collecting the content and description of static analysis rules formulated by experts, and initializing the static analysis rule knowledge base. The knowledge base includes the functional description, detailed explanation, and content of the static analysis rules. If a rule lacks a functional or detailed description, LLM is used to reverse-engineer the corresponding description based on the rule. After manual review and adjustment, the description is stored in the knowledge base." The specific method is as follows: The knowledge base includes a static analysis rule knowledge base and a static analysis tool knowledge base. Both knowledge bases use a hierarchical storage structure and each contains a raw data layer, a metadata layer, and a vector embedding layer.
[0062] The raw data layer of the static analysis rule knowledge base contains static analysis rules formulated by experts, along with functional descriptions of those rules. The metadata layer, extracted from the raw data based on LLM, includes information such as the applicable code defect categories, applicable programming languages and development frameworks, static analysis tools used, and corresponding static analysis methods.
[0063] The raw data layer of the static analysis tool knowledge base contains descriptive information about the static analysis tools, such as applicable programming languages, detectable defect types, and syntax specifications for static analysis rules. The metadata layer, extracted from the raw data based on LLM, includes information such as applicable programming languages, supported static analysis methods, and analysis types for the static analysis tools.
[0064] The vector embedding layers of both knowledge bases above are obtained by deduplicating and standardizing the original data content, and then embedding high-dimensional vectors based on a large model. The specific embedding method is as follows: First, the text is processed by a tokenizer and converted into a word sequence based on Byte Pair Encoding (BPE); then, word vectors are generated, with each word's vector consisting of three superimposed parts: E... out =E token +E position +E segment E token E is the vector to which it is mapped. position For position encoding, and E segment This serves as a marker for the sentence. The first two are necessary, while E... segmentThis is optional; if the large model used for text embedding does not support it, this part is unnecessary. After encoding through the Transformer layer, the hidden state of the last layer is taken as the word vector, and the i-th dictionary word vector is denoted as H. i After obtaining the word vectors, the sentence vector is calculated using the mean pooling method. Finally, the vector of each sentence is stored in the corresponding vector database. Furthermore, the embedding values of obviously unrelated samples can be added to the vector database for negative sampling, thereby effectively helping to distinguish "no-match queries".
[0065] The specific steps for "collecting historical software defect information, including correct / incorrect versions of the software source code, software defect descriptions, and repair information, and storing it in plain text" are as follows: From the historical software change information, determine whether a software change is a defect fix through keyword matching and other methods, and obtain the code change information used to fix the defect, which is then stored as historical defect information. This includes correct / incorrect versions of the software source code, software defect descriptions, and repair information, and is stored in plain text.
[0066] Step 2: Using LLM, based on the historical defect information collected in Step 1, generate rule descriptions, including the target programming language, applicable programming framework and software module, defect type and defect cause, matching algorithm used, matching mode, key variable types and semantic information, false positive cases to be excluded, etc.
[0067] The specific implementation method for step 2, "Using LLM to generate rule descriptions based on the historical defect information collected in step 1, including the target programming language, applicable programming framework and software module, defect type and cause, matching algorithm used, matching mode, key variable types and semantic information, and false positive cases to be excluded," is as follows: Combine correct and incorrect versions of software source code, descriptions of software defects, and software repair information into prompt words, input them into the LLM, and obtain the description for the static scanning rule. This description mainly includes two aspects: First, a description of the rule's function, such as the target programming language, applicable programming framework and software module, defect type and cause; second, the implementation details of the rule, including the matching algorithm used, matching mode, key variable types and semantic information, and false positive cases to be excluded. The generated static scanning rule description information is stored in plain text format in Markdown.
[0068] Step 3: Summarize and optimize the rule description based on LLM, generate query keywords and query statements, search the knowledge base of static analysis tools, and match appropriate static analysis tools based on factors such as the programming language the rule is oriented towards and the matching algorithm behind it, through semantic similarity.
[0069] The specific implementation method of step 3, "Based on LLM, according to the rule description, retrieve the knowledge base of static analysis tools, and match suitable static analysis tools based on semantic similarity according to the programming language targeted by the rules and the underlying matching algorithm," is as follows:
[0070] First, based on the static analysis rule description generated in step 2, the description content is summarized and optimized using LLM to generate query keywords and query statements. The static analysis tool knowledge base is then searched using a hybrid search strategy combining keyword similarity retrieval and semantic retrieval. The search process begins with keyword retrieval, searching and filtering key fields in the metadata, such as code defect types and the analysis tools used, to obtain records containing the keywords from the metadata layer of the knowledge base.
[0071] The subsequent vectorized retrieval implementation method is as follows: During the retrieval process, the cosine similarity between the query vector u and the vector v in the database is used. The matching process is performed, with higher similarity values indicating a better match. The data is then sorted from highest to lowest similarity, and the highest-similar normal data is used as the static analysis tool. If the returned results are empty, or if all of the top K most similar data (K is typically 3-5) are negative samples, the output "Unable to define rules, exit process" is displayed.
[0072] Step 4: Retrieve rules with similar functions or implementation details to the given rule from the knowledge base, and select the K rules with the highest similarity as minority cases. Combine the minority cases with information such as task description, rule description, type of rule detection tool, and static analysis rule syntax specifications to construct complete prompt words.
[0073] The specific implementation method of "retrieving rules with similar functions or implementation details based on the knowledge base and selecting the K most similar rules as few sample cases" in step 4 is as follows: Using a retrieval strategy similar to step 3, combining keyword similarity retrieval with semantic retrieval, a retrieval is performed in the static analysis rule knowledge base. First, the keyword similarity matching method is used to match fields in the metadata such as code defect categories, applicable programming languages and development frameworks, and static analysis tools and methods. For the keyword matching results, vectorized semantic matching is performed, embedding the target rule content into V. Rule The vectors are then used for similarity matching, and the similarity scores are reordered from highest to lowest. The descriptions and content of the top K rules with the highest similarity scores (usually K is 3 to 5) are used as the few sample cases.
[0074] In the phrase "combining a small number of cases with information such as task description, rule description, type of rule detection tool, and static analysis rule syntax specification to construct a complete prompt word", the "task description" is natural language text in the form of "please generate rules for the static analysis tool based on the rule description", while the "rule description" has already been generated above; the type of rule detection tool is obtained in step 3; and the "static analysis rule syntax specification" comes from the static analysis tool knowledge base.
[0075] Step 5: Generate analysis rules for the selected type of static analysis tool based on the large model, and initially verify the effectiveness of the rules through historical defects. If the rules are invalid, regenerate the rules and re-verify them. If the rules are still invalid after multiple regenerations, return to Step 2 to regenerate and adjust the rule descriptions, and continue with the subsequent processes.
[0076] In step 5, the specific implementation of "generating analysis rules for the selected type of static analysis tool based on the large model" is as follows: Select a pre-trained large language model without fine-tuning. Input the prompt words constructed in step 4 into the large model to generate the analysis rules for the static analysis tool, which are then stored as a text file. Subsequently, run the corresponding static analysis process. If the rules of the static analysis process are in the form of configuration files, such as YAML or XML, the corresponding scanning tool can be run directly to load the rules. If the rules of the static analysis process are in the form of executable files or dynamic link libraries, such as C++ or Scala, the generated rules need to be compiled before execution.
[0077] The specific implementation method for "preliminarily verifying the effectiveness of the rule through historical defects" is as follows: Use the generated rule to scan the source code of the historical defects that generated the rule, collect the generated scan logs, and analyze the scan results from the following three perspectives: ① Legality: Whether the scan can be completed normally without abnormal exits due to rule syntax errors, tool runtime errors, etc.; ② Completeness: Whether the corresponding type of defect can be found in the defective version code; ③ Low false positive rate: Whether the corresponding defect is not reported in the version after the defect is fixed. Only when all three conditions are met simultaneously can the rule be considered initially effective; otherwise, the rule is invalid.
[0078] The specific implementation method for "If the rule is invalid, you can first regenerate the rule and re-validate it. If the rule is still invalid after multiple regenerations, you need to return to step 2 to regenerate the rule description, adjust the rule description, and continue the subsequent process" is as follows: If the scan result is invalid, you can first try to regenerate the rule. Max Retry (usually) Max ≤3), if it exceeds Retry MaxIf the rule is still invalid after regeneration, it is necessary to manually analyze the errors, false positives and false negatives in the rule syntax, adjust the rule description, and re-execute step 3 and subsequent steps.
[0079] Step 6: Call the appropriate static analysis tool to scan for defects in the new version of the code based on the generated rules.
[0080] The specific steps for "calling the corresponding static analysis tool and scanning for defects in the new version of the code based on the generated rules" in step 6 are as follows: Call the corresponding static analysis tool, use the rules that have been initially verified to be effective, scan the entire source code of the project to be analyzed, and collect the defect reports generated by the tool.
[0081] Step 7: Evaluate the effectiveness of the rules based on the scan results. If the rule is effective, add it to the rule knowledge base; if the rule is ineffective, optimize the rule description based on the false positives / false negatives.
[0082] Step 7, "Evaluate the effectiveness of the rules based on the scan results. If the rule is effective, add it to the rule knowledge base; if the rule is invalid, optimize the rule description based on the false positives / false negatives," is implemented as follows: A rule is considered effective if the defect report generated by the analysis tool meets one of the following conditions: ① It generates a code defect report that has been verified and analyzed to be effective; ② The rule does not find any problems, and the number of false positives is less than the human threshold TH. MaxFP If the rule is valid, its description and content can be added to the rule knowledge base using the method in step 1; otherwise, if the rule is invalid, an attempt can be made to regenerate it. Max If the first attempt is still ineffective, the rule description can be adjusted and step 3 and subsequent steps can be re-executed to generate, run and evaluate the rule again.
[0083] The following analysis will focus on the Linux kernel source code (hereinafter referred to as Linux) to describe the specific implementation method of this application. The process is as follows: Figure 2 As shown. Linux is the world's most widely used open-source operating system, with extremely broad applications, and is a crucial cornerstone of modern information technology. The Linux kernel contains 30 million lines of C code, including not only general logic such as task scheduling and memory management, but also a large amount of C code developed by driver vendors. Therefore, it involves a great deal of domain knowledge and principles, making the manual formulation of static analysis rules a time-consuming and laborious task. Using the method proposed in this application, the workload of rule formulation can be greatly reduced for analysis. The specific application steps are as follows:
[0084] First, the knowledge base for static analysis tools was initialized. Semgrep, Python-Libclang, and Joern were selected. Semgrep is a rule-based code defect scanner; it takes YAML rules as input and scans the input code directly. Python-Libclang is a Python-based framework that allows customization of rules for C / C++ code. Joern is a Java and Scala-based code analysis framework that allows the creation of detection rules for multiple languages using Scala scripts. Descriptive information about these tools was collected from their official websites, including applicable programming languages, detectable defect types, and the syntax of static analysis rules. Metadata was extracted from these descriptions using LLM (Local Level Modeling) to include information such as applicable programming languages, supported static analysis methods, and analysis types. At the vector embedding layer, the original data was deduplicated and standardized before being embedded into a high-dimensional vector based on a large model.
[0085] The initial data layer of the static analysis rule knowledge base consists of static analysis rules defined by experts, along with their functional descriptions. The metadata layer, extracted from the initial data based on LLM (Local Level Management), includes information such as the applicable code defect categories, applicable programming languages and development frameworks, the static analysis tools used, and the corresponding static analysis methods. The remaining processes are essentially the same as those for the static analysis tool knowledge base.
[0086] In addition to initializing the knowledge base, historical defect information also needs to be collected. The code is fully cloned from the Linux software repository using Git tools to obtain all versions. If storage space is limited or network transmission speed is restricted, the number of commits pulled can be limited, such as retrieving the latest version and the most recent 5000 commits. Subsequently, commit information is queried using keyword matching and other methods to filter out a list of all commits that fix software defects (Bugfix). List The i-th defect is denoted as Bugfix. i Extract each bugfix i For each file that has a difference, there are correct and incorrect versions, as well as commit descriptions in the version control system. The commit descriptions usually include descriptions of the corresponding defects and information on how to fix them.
[0087] In the actual experiment, keywords such as "Misuse" and "Concurrency" were used to match in the Git repository to query various types of defects, including API misuse and concurrency defects. A total of 23 defects were collected, including 7 API misuses, 5 defects of using APIs after release, 6 concurrency defects, and 5 out-of-bounds access defects.
[0088] For each bugfix i The correct and incorrect versions of the software source code, descriptions of software defects, and software fix information are combined into Markdown plain text prompts. Then, by calling the large model APIs exposed by cloud service providers, such as the Qwen-2.5-72B model, a description of the static scanning rules, including rule functionality and implementation details, can be obtained. The rule functionality description mainly includes information such as the target programming language, applicable programming framework and software module, defect type, and defect cause. The implementation details include a natural language description of the matching algorithm used, matching mode, key variable types and semantic information, and false positive cases to be excluded. All of the above description information is stored in plain text format in Markdown, denoted as RuleDesc. i The rule to be generated is denoted as Rule. i .
[0089] Based on each static analysis rule description generated above, RuleDesc i Based on LLM, the description content is summarized and optimized to generate search metadata and a description of the capability requirements for static analysis tools. Based on the above content, the static analysis tool knowledge base is searched. First, keyword search is performed to search and filter key fields in the metadata, such as code defect type and analysis tools used, to obtain records with keywords in the metadata layer of the knowledge base.
[0090] The subsequent vectorized retrieval, based on cosine similarity, performs matching and sorts the data from highest to lowest similarity, selecting the normal data with the highest similarity as the generated rule. i The proposed static analysis tool is SATool. i If the returned result is empty, the highest similarity is lower than the set minimum similarity threshold, or all of the top K similarities (usually K is 3 to 5) are negative sampled data, then the output "Unable to formulate rules" will be displayed, and the process will exit.
[0091] In actual experiments, the large model successfully selected the appropriate static analysis tools for different defect types. For interface misuse issues, Semgrep was automatically selected; for concurrency issues involving locking, unlocking, and memory release followed by reuse, Joern was automatically selected; and for other issues, Python-LibClang was automatically selected.
[0092] Based on the selected static analysis tool SATool i and the defined rule description information RuleDesc iThe search retrieves similar rules as few-sample cases. Based on LLM, the description content is summarized and optimized to generate natural language descriptions for retrieval metadata and similar rule queries. A retrieval strategy combining keyword retrieval and semantic retrieval is used to search the static analysis rule knowledge base. First, keyword similarity matching is used to match fields in the metadata such as code defect categories, applicable programming languages and development frameworks, and static analysis tools and methods. For the keyword matching results, vectorized semantic matching is performed, embedding the target rule content into a vector. Similarity matching is then performed at the vectorization layer, and the rules are re-sorted from highest to lowest similarity. The top K rules (usually K is 3-5) with the highest similarity are used as few-sample cases. Subsequently, these few-sample cases are integrated with the task description, rule description, type of rule detection tool, and static analysis rule syntax specifications into plain text prompts. Each of these pieces of information is combined to construct the complete prompt for generating the rule, `RulePrompt`. i .
[0093] Select a pre-trained large language model and apply RulePrompt. i After inputting the large model, the generated analysis rules will be returned. i The large model is then returned to the data storage as a text file. For some large models, the generated code blocks may contain backticks that mark Markdown code blocks, so a script may need to be written to remove the backticks to avoid syntax errors. Subsequently, in the Bugfix... i The rules generated were run on both the flawed and fixed versions. For Semgrep YAML rules, Semgrep's config options were used to load the generated rules. For Python-LibClang and Joern, the rules were exported as Python and Scala files, respectively. The former required a Python 3 interpreter with LibClang installed, while the latter required the Joern program to execute the generated Scala script as an interpreter. Subsequently, Rules were collected. i The generated logs are scanned and analyzed. A rule can only be considered preliminarily effective if the tool simultaneously meets the following conditions: ① completes the scan normally without abnormally exiting due to rule syntax errors, tool runtime errors, etc.; ② can find the corresponding type of defect in the defective version of the code; ③ does not falsely report the corresponding defect in the version after the defect is fixed. Otherwise, the rule is invalid. If the rule is invalid, you can try generating a Retry loop again. Max Retry (usually) Max ≤3), if it exceeds Retry MaxIf the rule remains ineffective after a second regeneration, manual analysis is required to identify errors, false positives, and false negatives in the rule syntax. The rule description must then be adjusted, and step 3 and subsequent steps must be re-executed. In actual experiments, 17 rules could be preliminarily generated as valid rules with up to three direct retries, while the remaining rules were invalid at the grammatical level.
[0094] The appropriate static analysis tool is invoked, and using rules that have been preliminarily verified as effective, a full scan of the source code of the project to be analyzed is performed, and defect reports generated by the tool are collected. This list of reports is denoted as List. Defect List Defect (i, j) represents the i-th rule. i The j-th report generated.
[0095] List of defect reports generated by the report analysis tool Defect (i, j) can be determined as a rule when one of the following conditions is met. i It has shown good results in practical applications: ① It generates valid code defect reports that have been verified and analyzed in practice. ② Rule i Although no problems were found, the number of false alarms was less than the human threshold TH. MaxFP If the rule works well, its description and content can be added to the rule knowledge base; otherwise, try regenerating the rule first. Max If the first attempt is still ineffective, you can adjust the rule description and try to re-execute step 3 and subsequent steps to generate, run and evaluate the rule again.
[0096] In the actual experiment, the generated rules were used to scan the Linux kernel source code and set TH. MaxFP =50 contains two rules, which respectively reveal the following two types of problems, totaling 9 instances:
[0097] In the wireless network driver module, the variable 'vc' in mt7915 / mcu.c:1180 failed to be empty, and one instance was found.
[0098] The GPU driver amdgpu_userq_fence.c unsafely uses memdup_user to allocate array memory, and the array length uses an externally passed value. memdup_array_user should be used instead. Eight instances of this were found in this file.
[0099] Based on the above, the two types of rules mentioned above can be directly added to the knowledge base, while the remaining 12 rules that can be scanned normally but do not produce defects can also be added to the rule knowledge base.
[0100] In addition, nine rules failed to meet the requirements, including five consistently syntax errors and four rules with false alarms exceeding the threshold. For example, the rule used to detect the concurrency defect "locks must be released upon abnormal exit" generated over 500 alerts in the source code, far exceeding the threshold. MaxFP The threshold is so high that such fine-grained analysis code is practically impossible in real-world applications. Therefore, these invalid rules will not ultimately be added to the static analysis rule base.
[0101] This application presents an adaptive static analysis rule generation method based on RAG and LLM, which addresses the challenges of formulating domain- and business logic-related rules for static analysis in software, as well as the low accuracy of LLM rule generation due to the limited relevance of small sample suggestions. Through the above steps, static defect scanning rules can be effectively formulated based on historical error correction information, preventing the recurrence of historical defects and significantly improving software quality.
[0102] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0103] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0104] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0107] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for automatically generating static analysis rules based on LLM and RAG, characterized in that, include: Acquire information data; the information data includes static analysis rules and corresponding descriptive information stored in the knowledge base, as well as historical software defect information; LLM is used to generate rule description information based on the historical software defect information; Based on LLM, the system performs summarization and optimization based on the rule description information, and matches static analysis tools based on semantic similarity to determine the static analysis tools. Construct complete prompt words; The complete prompt includes: few sample case information, rule description information, type of static analysis tool, task description, and static analysis rule syntax specification information; the few sample case information is obtained by using a hybrid retrieval strategy that combines keyword similarity and semantic retrieval, performing keyword and vectorized semantic matching, sorting by similarity, and selecting a set number of static analysis rule contents and corresponding description information with high similarity values. Based on the pre-trained large language model, the analysis rules of the static analysis tool are determined according to the complete prompt words, and the rules are scanned and verified using software historical defect information to obtain the verification results. If the verification result indicates that the rule is invalid, the analysis rule is regenerated based on the pre-trained large language model. If the rule is invalid for a set number of times, the process returns to the step of "using LLM to generate rule description information based on the software's historical defect information". The prompt words for the LLM-generated rule description information are adjusted, and subsequent rule generation and scanning steps are executed until the verification result indicates that the rule is valid. Based on the analysis rules of static analysis tools, the corresponding static analysis tools are invoked to scan the source code of the project to be analyzed and the defect reports generated by the tools are collected. The results will be evaluated based on the defect report. If the evaluation result is valid, the static analysis rules and corresponding descriptive information of the static analysis tool will be added to the knowledge base. If the evaluation result is invalid, the analysis rules are first regenerated based on the pre-trained large language model. If there are invalid rules after a set number of attempts, the process returns to the step of "generating rule description information based on the software's historical defect information using LLM". The prompt words for generating the rule description information are then adjusted. After correcting the prompt word content, the subsequent steps are executed again until the evaluation result is valid.
2. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The knowledge base includes: a static analysis rule knowledge base and a static analysis tool knowledge base; both the static analysis rule knowledge base and the static analysis tool knowledge base use a hierarchical storage structure and each contains a raw data layer, a metadata layer and a vector embedding layer. The raw data layer of the static analysis rule knowledge base contains predefined static analysis rules and corresponding functional description information; the metadata layer of the static analysis rule knowledge base is extracted from the raw data based on LLM and contains information on the code defect categories to which the rules apply, the applicable programming languages and development frameworks, the static analysis tools used, and the corresponding static analysis methods. The raw data layer of the static analysis tool knowledge base contains descriptive information about the static analysis tools; the metadata layer of the static analysis tool knowledge base is extracted from the raw data based on LLM and contains information on the programming languages applicable to the static analysis tools, supported static analysis methods, and analysis types. The vector embedding layers corresponding to the static analysis rule knowledge base and the static analysis tool knowledge base are obtained by embedding high-dimensional vectors into a large model after deduplication and standardization of the original data content.
3. The method for automatically generating static analysis rules based on LLM and RAG according to claim 2, characterized in that, Embedding high-dimensional vectors based on large models involves specific embedding methods, including: The standardized original data text is converted into a word sequence using a word segmenter based on byte-pair encoding. After encoding using a Transformer layer, the hidden state of the last layer is selected as the word vector. The expression corresponding to each word vector is: AND out =And token +E position +E segment ; Among them, E token E is the vector to which it is mapped. position For position encoding; E segment E serves as a marker for sentences. out Word vectors; Based on the word vectors, the sentence vector is calculated using the mean pooling method; the sentence vector... H i Let be the word vector of the i-th word; T is the total number of words in the sentence; Store the vector of each sentence into the corresponding vector database.
4. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The software historical defect information includes: correct / incorrect version software source code, software defect descriptions and fix information; the software historical defect information is stored in plain text format.
5. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The rule description information includes: functional rule description information and implementation details; The functional rule description information includes: the target programming language, the applicable programming framework and software module, the defect type, and the defect cause; The implementation details include: the matching algorithm used, the matching mode, the types and semantic information of key variables, and the false positive cases that need to be excluded; The rule description information is stored in plain text format in Markdown.
6. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, Based on LLM, the system performs summarization and optimization based on the rule description information, and matches static analysis tools based on semantic similarity to determine the static analysis tools, specifically including: Based on the LLM, the rule description information is used to perform summarization and optimization to generate query keywords and query statements; Based on the query keywords and query statements, static analysis tools are matched using semantic similarity to determine the appropriate tools. A hybrid retrieval strategy combining keyword similarity and semantic retrieval is employed to search the static analysis tool knowledge base and match the relevant tools. This hybrid strategy includes: performing query keyword retrieval and filtering, as well as vectorized retrieval. The vectorized retrieval includes: using the cosine similarity between the vector u of the query statement and the vector v in the database. Perform a match; The data is sorted from highest to lowest similarity, and the matching data corresponding to the highest similarity is selected as the static analysis tool.
7. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The conditions for determining whether the verification result is a valid rule specifically include: Whether the scan can be completed normally without exiting abnormally due to rule syntax errors or tool runtime errors; Did you find the corresponding type of defect in the defective version of the code? And did you ensure that the corresponding defect was not reported in the version after the defect was fixed? 8. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The criteria for determining the validity of the evaluation results include: generating a valid code defect report after actual verification and analysis; no issues were found with the rules; and the number of false positives is less than a preset threshold TH. MaxFP .
9. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The analysis rules of static analysis tools are configured in the form of configuration files, including YAML and XML.
10. The method for automatically generating static analysis rules based on LLM and RAG according to claim 1, characterized in that, The analysis rules of static analysis tools are in binary form, including executable files and dynamic link libraries.