A large language model-based api misuse detection method and system

Abstract

The application discloses an API misuse detection method and system based on a large language model, and belongs to the technical field of software security. In order to solve the problem that API misuse detection in the prior art relies on manual rules and has a limited coverage, the application mainly generates API usage specifications by analyzing API source codes based on a large language model, verifies the correctness of the specifications by automatically constructing API calling codes that violate the specifications and combining dynamic execution and static analysis means, and then generates refined usage specifications and constructs detection codes to realize misuse identification. The application can automatically detect API misuse behaviors in application software, including null pointer dereferencing, memory leakage and double release, and the like, and improves the security and robustness of software.

Description

Technical Field

[0001] This invention belongs to the field of software security technology, specifically relating to an API misuse detection method and system based on a large language model. Background Technology

[0002] In this era of rapid information technology development, software has become deeply integrated into human life, becoming an indispensable part of modern society. Modern software is typically composed of numerous components, with complex structures and rich functionality. Reusing existing components can not only accelerate the development process but also reduce redundant development work. Some software libraries provide developers with ways to reuse components with specific functions through Application Programming Interfaces (APIs). However, because API users often lack understanding of the internal implementation details of APIs and do not have a sufficient understanding of the correct usage methods, API misuse may occur, leading to serious software security problems, such as system crashes caused by null pointer dereferences, service unavailability issues, resource exhaustion due to memory leaks, and denial-of-service problems.

[0003] To ensure the correct use of APIs and improve software security, security experts develop API usage guidelines based on existing security knowledge and detect API misuses that violate these guidelines. However, manually generating API usage guidelines for a large number of APIs is both difficult and impractical, and manual analysis may lead to errors or omissions due to the limitations of expert knowledge in security guidelines. Therefore, automating the generation of API usage guidelines and the detection of corresponding API misuses is essential. Summary of the Invention

[0004] The purpose of this invention is to provide an API misuse detection method and system based on a large language model. This method can automatically generate API usage specifications based on the knowledge of the large language model, detect API misuse in application software, including null pointer dereferences, memory leaks, and double releases, thereby improving software security.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A method for detecting API misuse based on a large language model includes the following steps: Initial API usage specifications are generated based on the source code and documentation information of the target software library; Based on the initial API usage specification, the large language model is guided to generate positive and negative call codes, and the positive and negative call codes are executed for verification and consistency analysis to obtain the verified API usage specification. The execution results of the negative call code are clustered, and the clustering results guide the large language model to complete the information of the verified API usage specifications, so as to obtain refined API usage specifications. The refined API usage specifications are integrated with preset misuse pattern templates to generate a detection script for the target application, and the detection script is run to identify API misuse behaviors in the target application.

[0007] Furthermore, based on the source code and documentation information of the target software library, an initial API usage specification is generated, including: Extract the API list from the official documentation of the target software library, and parse the source code of each API based on the abstract syntax tree; The API function definition, calling method, and parameter description are embedded into a standardized prompt template to generate instantiation prompts for the target API. Input the instantiated prompt words into the large language model to obtain the generated initial API usage specifications and corresponding examples of non-compliant code.

[0008] Furthermore, the execution verification and consistency analysis of the positive and negative call code are performed, including: Execute the forward call code and invoke the automated repair program based on the runtime exception information until the correct code that has been successfully executed is obtained; By comparing the differences in API call location and parameter modifications between the negative call code and the correct code, the semantic consistency between the negative call code and the initial API usage specification is verified. If a negative call that has passed consistency verification generates an API runtime error, it is determined that the corresponding initial API has passed the specification verification.

[0009] Furthermore, based on the runtime anomaly information, an automated repair procedure is invoked, including: Construct repair prompts based on code compilation error messages or execution failure messages; The repair prompts are fed back to the large language model to update the original code context and generate the repaired positive call code; The repaired forward call code is executed repeatedly, and the repair is guided based on the feedback information until the code executes successfully.

[0010] Furthermore, the execution results of the negative call code are clustered, including: Analyze the execution log of the negative call code during its execution process to extract error description information and call stack information; Negative call codes with the same error type and similar error location are grouped into the same call behavior category.

[0011] Furthermore, based on the clustering results, the large language model is guided to complete the information for the validated API usage specifications, including: Based on the aforementioned call behavior category, construct refined prompt words that include call parameters, error triggering conditions, and execution context information; The refined prompt words are input into a large language model to generate refined API usage specifications labeled with potential problem points and trigger constraints.

[0012] Furthermore, before guiding the large language model to complete the information for the verified API usage specifications based on the clustering results, the process also includes: The execution log is filtered using preset heuristic rules to identify error messages unrelated to the API and extract API-related errors that match the target API call location.

[0013] Furthermore, the refined API usage specifications are integrated with preset misuse pattern templates to generate a detection script for the target application, including: Build code templates for detecting null pointers, memory out-of-bounds errors, and dereference errors; The refined API uses the API parameters and calling format specified in the specification to fill the detection code template, generating a detection script with complete detection logic.

[0014] Furthermore, running the detection script to identify API misuse behavior in the target application includes: The detection script is loaded using a static analysis engine, and all API call points in the target application are scanned to identify misuse points that violate the refined API usage guidelines.

[0015] An API misuse detection system based on a large language model includes: The API usage specification generation module is used to generate initial API usage specifications based on the source code and documentation information of the target software library. The API usage specification correctness verification module is used to guide the large language model to generate positive and negative call code based on the initial API usage specification, and to perform execution verification and consistency analysis on the positive and negative call code to obtain the verified API usage specification. The API usage specification refinement module is used to cluster the execution results of the negative call code, and guide the large language model to complete the information of the verified API usage specification based on the clustering results, so as to obtain the refined API usage specification. The API misuse detection module is used to integrate the refined API usage specifications with the preset misuse pattern templates to generate a detection script for the target application, and run the detection script to identify API misuse behaviors in the target application.

[0016] The beneficial effects achieved by this invention are as follows.

[0017] 1. This invention utilizes a large language model to automatically analyze API source code, generating API usage specifications in various forms that combine natural language descriptions and code examples, thereby improving the efficiency and accuracy of API usage specification generation.

[0018] 2. This invention guides a large language model to automatically generate code that violates API usage specifications by designing prompt words, and combines this with a dynamic execution analysis mechanism to verify the correctness of API usage specifications, thus avoiding the subjectivity and inefficiency of manual verification.

[0019] 3. This invention combines clustering and common operation analysis mechanisms to refine existing API usage specifications, generating more specific and targeted API usage specifications, thereby improving the coverage and accuracy of subsequent detection code.

[0020] 4. This invention automatically constructs detection code through a templated detection mechanism, enabling static analysis and detection of software using APIs, and can automatically identify various types of API misuse issues that violate API usage specifications.

[0021] 5. This invention supports the detection of various common API misuse issues, including null pointer dereferencing, memory leaks, double freeing, etc., improving the ability to identify potential software security risks and enhancing software stability and security.

[0022] 6. This invention introduces mechanisms such as automated consistency verification and heuristic path analysis, which effectively improves the controllability of the generated code of the large model and the accuracy of misuse judgment, thereby enhancing the overall detection reliability and practicality of the system. Attached Figure Description

[0023] Figure 1 A flowchart illustrating an API misuse detection method based on a large language model, provided as an example. Figure 2 Flowchart for verifying the correctness of API usage specifications; Figure 3 A detailed flowchart for API usage specifications; Figure 4 This is a framework diagram of an API misuse detection system based on a large language model, provided for an embodiment. Detailed Implementation

[0024] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0025] This invention provides a method for detecting API misuse based on a large language model, such as... Figure 1 As shown, the standardized processing flow is explained below: Step S1: Generate an initial API usage specification (APSR) based on the source code and documentation information of the target software library.

[0026] Specifically, based on the library code content and its documentation, a preliminary API usage specification is generated using a large language model.

[0027] In an optional embodiment of the present invention, step S1 may include: Step S11: Extract the API list from the official documentation of the target software library and obtain the source code content of each API based on the abstract syntax tree. Step S12: Embed the API function definition, calling method, and parameter description into the standardized prompt word template to generate an instantiation prompt word for the target API; Step S13: Input the instantiated prompt word into the large language model to obtain the generated initial API usage specifications and corresponding violation code examples.

[0028] Step S2: Based on the initial API usage specification, guide the large language model to generate positive and negative call codes, and perform execution verification and consistency analysis on the positive and negative call codes to obtain the verified API usage specification.

[0029] Specifically, the original API usage specifications are dynamically validated to confirm their correctness. The validation process is as follows: Figure 2 As shown.

[0030] In an optional embodiment of the present invention, step S2 may include: Step S21: Execute the forward call code and call the automated repair program according to the running exception information until the correct code that was successfully executed is obtained; Step S22: Compare the differences in modifications at the API call location and parameter level between the negative call code and the correct code to verify the semantic consistency between the negative call code and the initial API usage specification; wherein, the consistency verification logic analyzes the key operations specified in the API usage specification and their relative positions and parameters. If the negative code and the correct code are consistent at the position and parameter level, it is confirmed that the generated non-compliant code conforms to the specification definition. Step S23: Execute the negative call code that has passed the consistency verification. If an API runtime error occurs, it means that the current API usage specification has actual detection capability, and the corresponding initial API usage specification verification is deemed to have passed.

[0031] Specifically, step S21, which involves invoking the automated repair program based on the runtime anomaly information, may further include: Step S211: Construct a repair prompt word based on the code compilation error message or execution failure message; Step S212: Feedback the repair prompt words to the large language model to update the original code context and guide the generation of repaired positive call code that meets the constraints; Step S213: Execute the repaired forward call code repeatedly and guide the repair based on feedback information until the code executes successfully.

[0032] Step S3: Cluster the execution results of the negative call code, and guide the large language model to complete the information of the verified API usage specifications based on the clustering results, so as to obtain refined API usage specifications.

[0033] Specifically, the verified API usage specifications are further refined to improve their expressiveness and detection accuracy. The process is as follows: Figure 3 As shown.

[0034] In an optional embodiment of the present invention, step S3 may include: Step S31: Analyze the execution log of the negative call code during execution, and extract error description information and call stack information; Step S32: Negative call codes with the same error type and similar error occurrence locations are classified into the same call behavior category; Step S33: Construct refined prompt words containing call parameters, error triggering conditions, and execution context information based on the call behavior category; Step S34: Input the refined prompt words into the large language model to generate refined API usage specifications labeled with potential problem points and trigger constraints.

[0035] In an optional embodiment of the present invention, before step S33, the following may be included: The execution log is filtered using preset heuristic rules to identify error messages unrelated to the API and extract API-related errors that match the target API call location. If an error occurs at the target API call point and matches the call context, it is retained for specification refinement.

[0036] Step S4: Integrate the refined API usage specifications with the preset misuse pattern template to generate a detection script for the target application, and run the detection script to identify API misuse behaviors in the target application.

[0037] Specifically, based on validated and refined API usage guidelines, it enables the detection of misuse of API call behavior in applications.

[0038] In an optional embodiment of the present invention, step S4 may include: Step S41: Construct a detection code template for null pointer exceptions, memory out-of-bounds errors, and dereference errors; Step S42: Fill the detection code template with the API parameters and calling format in the refined API usage specification, and automatically merge them with the specific API parameters to generate a CodeQL detection script with complete detection logic; Step S43: Load the CodeQL detection script using the static analysis engine and scan all API call points in the target application to identify misuse points that violate the fine-grained API usage specifications.

[0039] This invention also provides an API misuse detection system based on a large language model, such as... Figure 4 As shown, the method described above includes: The API usage specification generation module is used to generate initial API usage specifications based on the source code and documentation information of the target software library. The API usage specification correctness verification module is used to guide the large language model to generate positive and negative call code based on the initial API usage specification, and to perform execution verification and consistency analysis on the positive and negative call code to obtain the verified API usage specification. The API usage specification refinement module is used to cluster the execution results of the negative call code, and guide the large language model to complete the information of the verified API usage specification based on the clustering results, so as to obtain the refined API usage specification. The API misuse detection module is used to integrate the refined API usage specifications with the preset misuse pattern templates to generate a detection script for the target application, and run the detection script to identify API misuse behaviors in the target application.

[0040] To verify the effectiveness and superiority of the method of the present invention, the following experimental tests were conducted: I. Effects of APSR generation method.

[0041] 1. Experimental Dataset: 25 APIs were randomly selected from each of 8 typical software libraries, totaling 200 APIs. Through manual analysis of their documentation, source code, and APSRs generated by other tools, 404 APSRs were obtained from the 200 APIs, which served as the evaluation dataset.

[0042] 2. Experimental Results: The results were evaluated using three metrics: precision, recall, and F1 score. With an average cost of only $0.12 per API, the method generated 311 APSRs, achieving a precision of 92.3% and a recall of 71.0%. Detailed results for each software library are shown in Table 1.

[0043] Table 1. APSR generation results for each software library As shown in Table 1, the method of the present invention can generate high-quality APSRs at a low cost, with high accuracy and good coverage.

[0044] II. API misuse detection effectiveness.

[0045] The method of this invention was applied to 47 real-world software projects, detecting 210 previously unknown API misuses with an accuracy rate of 77.2%. This result demonstrates that the method of this invention can effectively identify API misuse issues in real-world software.

[0046] III. Comparative Experiment.

[0047] 1. Experimental Dataset: A dataset containing defects and their corresponding APSRs was constructed. The sources of defects included: (1) defects reported in existing studies; and (2) defects detected by the tools of this invention. The final experimental dataset consisted of a total of 306 defects and their corresponding 58 APSRs.

[0048] 2. Experimental Results: The method of this invention was compared with three advanced tools (Advance, IPPO, and Goshawk). The results show that the method of this invention significantly outperforms existing tools in both the number of APSRs generated and the number of defects detected, as detailed in Table 2.

[0049] Table 2 Comparison results with existing tools As shown in Table 2, the method of the present invention can generate more effective APSRs and detect significantly more defects, demonstrating its obvious advantages in API misuse detection.

[0050] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for detecting API misuse based on a large language model, characterized in that, Includes the following steps: Initial API usage specifications are generated based on the source code and documentation information of the target software library; Based on the initial API usage specification, the large language model is guided to generate positive and negative call codes, and the positive and negative call codes are executed for verification and consistency analysis to obtain the verified API usage specification. The execution results of the negative call code are clustered, and the clustering results guide the large language model to complete the information of the verified API usage specifications, so as to obtain refined API usage specifications. The refined API usage specifications are integrated with preset misuse pattern templates to generate a detection script for the target application, and the detection script is run to identify API misuse behaviors in the target application.

2. The method as described in claim 1, characterized in that, Based on the source code and documentation information of the target software library, an initial API usage specification is generated, including: Extract the API list from the official documentation of the target software library, and parse the source code of each API based on the abstract syntax tree; The API function definition, calling method, and parameter description are embedded into a standardized prompt template to generate instantiation prompts for the target API. Input the instantiated prompt words into the large language model to obtain the generated initial API usage specifications and corresponding examples of non-compliant code.

3. The method as described in claim 1, characterized in that, The execution verification and consistency analysis of the positive and negative call code include: Execute the forward call code and invoke the automated repair program based on the runtime exception information until the correct code that has been successfully executed is obtained; By comparing the differences in API call location and parameter modifications between the negative call code and the correct code, the semantic consistency between the negative call code and the initial API usage specification is verified. If a negative call that has passed consistency verification generates an API runtime error, it is determined that the corresponding initial API has passed the specification verification.

4. The method as described in claim 3, characterized in that, Based on the runtime error information, an automated repair program is invoked, including: Construct repair prompts based on code compilation error messages or execution failure messages; The repair prompts are fed back to the large language model to update the original code context and generate the repaired positive call code; The repaired forward call code is executed repeatedly, and the repair is guided based on the feedback information until the code executes successfully.

5. The method as described in claim 1, characterized in that, Clustering the execution results of the negative call code includes: Analyze the execution log of the negative call code during its execution process to extract error description information and call stack information; Negative call codes with the same error type and similar error location are grouped into the same call behavior category.

6. The method as described in claim 5, characterized in that, Based on the clustering results, the large language model is guided to complete the information for the validated API usage specifications, including: Based on the aforementioned call behavior category, construct refined prompt words that include call parameters, error triggering conditions, and execution context information; The refined prompt words are input into a large language model to generate refined API usage specifications labeled with potential problem points and trigger constraints.

7. The method as described in claim 5, characterized in that, Before guiding the large language model to complete the information for the validated API usage specifications based on the clustering results, the process also includes: The execution log is filtered using preset heuristic rules to identify error messages unrelated to the API and extract API-related errors that match the target API call location.

8. The method as described in claim 1, characterized in that, The refined API usage specifications are integrated with preset misuse pattern templates to generate a detection script for the target application, including: Build code templates for detecting null pointers, memory out-of-bounds errors, and dereference errors; The refined API uses the API parameters and calling format specified in the specification to fill the detection code template, generating a detection script with complete detection logic.

9. The method as described in claim 1, characterized in that, Running the detection script to identify API misuse behaviors in the target application includes: The detection script is loaded using a static analysis engine, and all API call points in the target application are scanned to identify misuse points that violate the refined API usage guidelines.

10. An API misuse detection system based on a large language model, characterized in that, include: The API usage specification generation module is used to generate initial API usage specifications based on the source code and documentation information of the target software library. The API usage specification correctness verification module is used to guide the large language model to generate positive and negative call code based on the initial API usage specification, and to perform execution verification and consistency analysis on the positive and negative call code to obtain the verified API usage specification. The API usage specification refinement module is used to cluster the execution results of the negative call code, and guide the large language model to complete the information of the verified API usage specification based on the clustering results, so as to obtain the refined API usage specification. The API misuse detection module is used to integrate the refined API usage specifications with the preset misuse pattern templates to generate a detection script for the target application, and run the detection script to identify API misuse behaviors in the target application.

Images