RESTful APIs test method based on large language model

Abstract

The invention discloses a large language model-based RESTful APIs test method, which comprises the steps of large model natural language processing, RESTful APIs automatic black box test, OpenAPI document analysis, test sequence generation, sequential execution of operations in the test sequence, and finally execution of test cases. According to the technical scheme, the OpenAPI document and the natural language information fed back by the RESTful APIs in the testing process are analyzed through the large language model, key information such as parameter constraint and operation dependence is continuously and dynamically adjusted to optimize generation of the test case, the natural language information in the RESTful APIs test can be fully utilized through the large language model, and the test efficiency is improved. The method has the advantages of being capable of accurately judging the dependency relation and the constraint relation of the parameters, providing high-quality parameter values, obtaining higher operation coverage rate in the same test predication, and being stable in system fault triggering.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more particularly to a method for testing RESTful APIs based on a large language model. Background Art

[0002] In today's network era, modern Web services are booming at an unprecedented speed. Among many API architectural styles, REST (Representational State Transfer) has occupied a dominant position in the field of complex Web service design with its unique advantages and has gradually become the de facto standard for modern Web service design and implementation. With the in-depth and extensive adoption of RESTful APIs in building modern enterprise systems, ensuring the functional correctness of such APIs has become a crucial task, and a perfect testing process is very important for ensuring system quality.

[0003] The black box testing of RESTful APIs relies on the OpenAPI document as input and can automatically generate test cases to test RESTful APIs, greatly saving testing costs. However, the current testing method only utilizes the machine-readable part in the OpenAPI document, and only a few methods help with testing through the natural language information in the document. At the same time, almost all current research ignores the natural language information of RESTful APIs themselves. The insufficient utilization of natural language information restricts the effectiveness of current black box testing tools. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for testing RESTful APIs based on a large language model to solve the problems raised in the above background art.

[0005] To solve the above technical problems, the present invention provides the following technical solution: A method for testing RESTful APIs based on a large language model, the RESTful APIs testing method is an automated black box testing method that only relies on the OpenAPI document to automatically generate test cases and test RESTful APIs without any other additional input. The RESTful APIs testing method includes the following steps: S1: Parse the OpenAPI document and extract all the information required for testing into a list ; Preferably, the list is a list containing all the required information in the OpenAPI document, and each is a predefined data structure for storing operation information and parameter information.

[0006] S2: Generate test sequence; Preferably, generating the test sequence comprises: when generating for the first time, requesting the large language model to generate a sequence containing all Test sequence ; During the second and subsequent test sequence generation processes, test sequences are performed through operations that were not successfully tested Generation of; never successfully tested operations Randomly select an operation ; for the operation Generate test sequence ; Leverage extracted dependencies Test operations that never succeed; After all operations in the list reach the test number threshold, a random Select an operation to test.

[0007] S3: Execute the operations in the test sequence sequentially, for each operation to be tested Generate a batch of test cases ; Preferably, generating a test case includes: Parameter dependency extraction: The parameter dependency extraction process is dynamic. According to the server response in the test, the response information is provided to the large language model for parameter dependency extraction. The extraction is performed through two rounds of questions using a chain prompt method. Preferably, the operation to be tested is performed using a large language model. Parameters in , use a large language model to extract dependencies; extract whether these parameters depend on certain parameters in the operations that have been executed, and get a dependency dictionary , indicating the operation to be tested Parameters Point to the executed operation Response parameters .

[0008] Parameter constraint extraction: Use IDL as the constraint description language and use a large language model to extract constraint relationships; Preferably, the initial extraction uses the description field in the OpenAPI document to extract, collects natural language information in the server response during the test, and re-extracts and updates the constraint relationship after a set time or a certain amount of collected information. After extracting the constraints, the operation to be tested is obtained. The constraint relationship of the parameters .

[0009] Parameter Selection and Assignment.

[0010] Preferably, select parameters according to the test phase, and set the original mandatory parameters and the parameters in the constraint relationship as important parameters. When the operation fails to execute successfully, select important parameters. When retesting the operation after successful execution, select important parameters and randomly select the remaining parameters. After selection, obtain the parameter list for this test ; Assign values to the parameters in the , generate a candidate value range for each parameter, and the value range consists of the following values: (1) Dependent value: For the analyzed parameter dependency relationship, extract the dependent parameter values from the previously saved server responses, and preferably extract the parameter values from the latest server response; (2) Default value: The default value recorded for the parameter in the document; (3) Example value: Example values are divided into two types. One is the example value recorded in the document, and the other is the example generated by the large language model; (4) Random value: A value randomly generated according to the parameter type; Randomly select parameter values from the value range of each parameter to form a non-repeating , and add it to And delete the test cases that do not meet .

[0011] S4: Execute the test cases; Preferably, for each operation to be tested 's test case set , execute the test cases in sequence, generate HTTP requests and receive the response information returned by the server, including correct response information and error response information. Save the latest n correctly executed results. According to the successful execution results, verify the dependency relationship and the constraint relationship, and save the correct relationship for subsequent use; Preferably, perform error information processing, maintain a corresponding message list for each operation , save all unique error information in the message list. When receiving an error response message, remove the parameter values and timestamp information, and compare it with all the error information in the message list. When the cosine similarity with all existing messages is less than the set threshold, it is considered a new error message and added to the message list.

[0012] S5: When the test time is less than the set time budget, repeat the test process of S2 - S4.

[0013] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: Through the large language model, we can make full use of the natural language information in RESTful APIs testing, accurately judge the dependency relationship and constraint relationship of parameters, and at the same time provide high-quality parameter values. Achieve higher operation coverage and stable system fault triggering within the same test budget. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings: Figure 1 is a schematic diagram of the specific working process of the LLM-REST method provided by the embodiment of the present invention; Figure 2 is a schematic diagram of the test case generation algorithm provided by the embodiment of the present invention; Figure 3 is an example diagram of the operation of all tasks provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] To further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following combines the accompanying drawings and preferred embodiments to detail the specific implementation manners, structures, features and their effects according to the present invention as follows.

[0016] The embodiments of the present invention combine Figures 1 to 3 , and provide the following technical solution: A method for testing RESTful APIs based on a large language model. The RESTful APIs testing method is an automated black box testing method for RESTful APIs, which can automatically generate test cases and test RESTful APIs only relying on the OpenAPI document without other additional inputs. Exemplarily, LLM-REST includes the following steps: S1: Parse the OpenAPI document and extract all the information required for testing into a list ; Exemplarily, is a list containing all the required information in the OpenAPI document, and each is a predefined data structure for storing operation information, parameter information, etc.

[0017] S2: Generate a test sequence; Exemplarily, due to the design characteristics of RESTful APIs, there are dependency relationships between operations, and the operations need to be executed in sequence. By requesting the large language model to generate a list containing all Test sequence ; Further, during the generation of the second and subsequent test sequences, by focusing on the operations that have not been successfully tested, an operation will be randomly selected from the operations that have not been successfully tested and a test sequence will be generated for it . When generating, we also consider the dependency relationship, and at this time, the dependency relationship extracted in the subsequent steps can be utilized ; If some operations cannot be successful all the time, the method will keep testing the operations and cannot explore the parameters of other operations. Therefore, after all the operations in reach the test times threshold, further operations will be randomly selected from for testing, where the threshold is set to .

[0018] S3: Execute sequentially the operations in, and generate a batch of test cases for each operation to be tested ; In this embodiment, as shown in combination with Figure 2 , which is the algorithm for generating test cases, specifically includes the following steps: S31: Parameter dependency extraction; Exemplarily, use a large language model to extract parameter dependencies, for the parameters in the operation to be tested , use a large language model to extract the dependency relationship, and further combine with the hint design in, and extract whether these parameters depend on some parameters in the operations that have been executed in two steps to obtain a dependency relationship dictionary Figure 3 , indicating that the parameters of the operation to be tested point to the response parameters of the executed operation . Here, the extraction process is dynamic, and response information is provided to the large language model for extraction according to the server response in the test. The aggregation hint design is as shown in . In parameter dependency extraction, the method of chained hints is used, and through two rounds of questioning, the accuracy of parameter dependency extraction is improved Figure 3 .

[0019] S32: Parameter constraint extraction; ​​Exemplarily, the IDL language is used as the constraint description language, and the large language model is used to extract the constraint relationships. For the initial extraction, the description fields in the OpenAPI document are used for extraction. During the testing process, the natural language information in the server response will be collected. After a certain period of time or when the collected information reaches a certain quantity, the extraction will be redone to update our constraint relationships. Further, after extracting the constraints, the operations to be tested are obtained. The constraint relationships of the parameters .

[0020] S33: Parameter Selection and Assignment; Exemplarily, according to the testing phase, parameters are selected. The original mandatory parameters and the parameters in are set as important parameters. When the operation fails to execute successfully, important parameters are selected. If the operation has been executed successfully, then when testing the operation again, important parameters are selected and the remaining parameters are randomly selected. After selection, the parameter list for this test is obtained. ; Furthermore, for the parameters in, values are assigned. A candidate value range is generated for each parameter. The value range consists of the following values: (1) Dependent values: For the analyzed parameter dependency relationships, the dependent parameter values are extracted from the previously saved server responses, and the parameter values in the latest server response are preferentially extracted; (2) Default values: The default values recorded for the parameters in the document; (3) Example values: Example values are divided into two types. One is the example values recorded in the document, and the other is the examples generated by the large language model; (4) Random values: Values randomly generated according to the parameter type.

[0021] Finally, parameter values are randomly selected from the value range of each parameter to form a non - repeating , and added to and test cases that do not meet are deleted.

[0022] S4: Test Case Execution; Exemplarily, for each set of test cases for the operations to be tested , the test cases are executed in sequence, HTTP requests are generated, and the response information returned by the server is received, including correct response information and error response information. The latest 10 correctly executed results are saved; According to the successful execution results, the dependency relationships and constraint relationships are verified, and the correct relationships are saved for subsequent use; Error Information Handling: Maintain a corresponding message list for each operation , save all unique error messages in the message list. When receiving an error response message, remove information such as parameter values and timestamps, and then compare it with all error messages in the message list. If the cosine similarity with all existing messages is less than , it is considered a new error message and added to the message list.

[0023] S5: If the test time is less than the set time budget, which is usually 1 hour, repeat the test process of S2 - S4 above.

[0024] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments with equivalent changes within the scope of the technical solution of the present invention. However, as long as it does not depart from the content of the technical solution of the present invention, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A testing method for RESTful APIs based on large language models, characterized in that: The RESTful APIs testing method is an automated black box testing method that relies solely on the OpenAPI document to automatically generate test cases and test RESTful APIs without any other additional input; The RESTful APIs testing method includes the following steps: S1: Parse the OpenAPI document and extract all the information required for testing into a list ; S2: Generate a test sequence; S3: Sequentially execute the operations in the test sequence, for each operation to be tested Generate a batch of test cases ; S4: Execute test cases; S5: When the test time is less than the set time budget, repeat the test process of S2 - S4.

2. The method for testing RESTful APIs based on a large language model according to claim 1, wherein: The said list is a list containing all the required information in the OpenAPI document, and each is a predefined data structure for storing operation information and parameter information.

3. The method for testing RESTful APIs based on a large language model according to claim 2, wherein: The generation of the test sequence includes: When generating for the first time, request the large language model to generate a test sequence containing all of ; During the generation of the second and subsequent test sequences, the test sequences are generated by operations that have not been successfully tested. are generated.

4. The method for testing RESTful APIs based on a large language model according to claim 3, characterized in that: Specifically included in the process of generating the test sequence for the second and subsequent times: Operations that have never been successfully tested Randomly select an operation from ; For the operation Generate a test sequence ; Construct a test sequence using the extracted dependencies ; Operations without successful tests After all operations in reach the test count threshold, an operation is randomly selected from the list for testing.

5. The method for testing RESTful APIs based on a large language model according to claim 4, wherein: The generation of a batch of test cases specifically includes: Parameter dependency extraction: The parameter dependency extraction process is dynamic. Response information is provided to the large language model based on the server response during the test for parameter dependency extraction. The extraction is carried out in a chained prompting manner through two rounds of questioning; Parameter constraint extraction: Use the IDL language as the constraint description language and use the large language model to extract constraint relationships; Parameter selection and assignment.

6. The method for testing RESTful APIs based on a large language model according to claim 5, characterized in that: The parameter dependency extraction is as follows: By using a large language model, for the operation to be tested parameters , use the large language model to extract the dependency relationship; determine whether these parameters depend on certain parameters in the operations that have been executed to obtain a dependency dictionary , indicating the operation to be tested parameters pointing to the response parameters of the executed operation .​ 7. A method for testing RESTful APIs based on a large language model according to claim 6, characterized in that: The parameter constraint extraction is as follows: for the initial extraction, the description fields in the OpenAPI document are used for extraction. During the testing process, the natural language information in the server response is collected, and after a set time or when the collected information reaches a certain quantity, the constraint relationship is re-extracted and updated. After extracting the constraints, the parameter constraint relationship of the operation to be tested is obtained. of the parameter .

8. A method for testing RESTful APIs based on a large language model according to claim 7, characterized in that: The parameter selection and assignment are as follows: Select parameters according to the test phase, and set the original mandatory parameters and the parameter in the constraint relationship as important parameters. When the operation fails to execute successfully, select important parameters. When testing the operation again after successful execution, select important parameters and randomly select the remaining parameters. After selection, a list of parameters selected for this test is obtained ; Assign values to the parameters in and generate a candidate value range for each parameter. The value range consists of the following values: (1) Dependency value: For the analyzed parameter dependency relationship, extract the dependent parameter value from the previously saved server response, and preferentially extract the parameter value from the latest server response; (2) Default value: The default value recorded for the parameter in the document; (3) Example value: The example value is divided into two types. One is the example value recorded in the document, and the other is the example generated by the large language model; (4) Random value: A value randomly generated according to the parameter type; Randomly select parameter values from the value range of each parameter to form non-repeating , and add them to And delete the test cases that do not meet .

9. A method for testing RESTful APIs based on a large language model according to claim 8, characterized in that: The execution of test cases includes: for each operation to be tested a set of test cases , execute the test cases in sequence, generate HTTP requests and receive response information returned by the server, including correct response information and error response information, save the latest n correctly executed results, verify the dependency relationship and the constraint relationship according to the successfully executed results, and save the correct relationships for subsequent use.

10. The method for testing RESTful APIs based on a large language model according to claim 9, wherein: The execution of the test case further includes: performing error message processing, maintaining a corresponding message list for each operation , saving all unique error messages in the message list. When a wrong response message is received, the parameter value and the timestamp information are removed and compared with all the error messages in the message list. When the cosine similarity with all existing messages is less than the set threshold, it is considered a new error message and added to the message list.

Images