A method and system for generating API test cases

Abstract

The present invention discloses a method and system for generating API test cases. The method comprises obtaining an invalid test request and its corresponding error response request; obtaining prompt information from the error response request; and modifying the error parameters in the invalid test request using the prompt information to generate a new test request. The solution provided by the present invention performs targeted corrections on the error parameters in the invalid test request, rather than arbitrarily generating based on OpenAPI description document information, so that a valid test request can be generated with a higher probability, thereby improving the ability to generate valid test cases.

Description

Technical Field

[0001] The present invention relates to the field of fuzzy testing technology, and in particular to a method and system for generating API test cases. Background Art

[0002] In recent years, with the rise of cloud computing and microservice architecture, the number of Web APIs used has shown a huge growth trend. Web API is a way for applications to provide services to the outside world. The most popular type of modern Web API is RESTful API, which is a type of Web API that follows the REST architectural style. Many well-known web service providers (for example, Google, Microsoft, Wordpress, etc.) provide digital services and assets through RESTful API. Due to the widespread use of RESTful API on the Internet, the security of RESTful API has also received widespread attention in recent years. Therefore, it is very important to study automated testing technology for RESTful API.

[0003] The key to testing RESTful APIs is how to generate effective test cases, explore the deep logic of the program, and discover program bugs. Effective test cases are composed of effective test HTTP requests, and the main challenge in generating effective test HTTP requests is how to generate legal values ​​for the parameters in the request. Most RESTful API test requests contain multiple request parameters. If a few of the request parameter values ​​are illegal, the test request will be rejected by the test target service, that is, an invalid test request. In the existing test case generation method, when each test request is generated based on the request template, specific values ​​are generated for all its request parameters. If individual parameter values ​​in the generated test request are illegal, resulting in an error response request after it is sent for testing, the existing method will discard the invalid request and generate a new test request based on the request template next time, that is, re-instantiate all parameter values ​​in the request template. On the one hand, such a test case generation method ignores the legal parameter values ​​in the invalid requests that have been tested. The next time a test request is generated, these legal data will not be used, and illegal new values ​​may be regenerated for these parameters. On the other hand, the illegal parameter values ​​in the invalid requests will not be modified in a targeted manner. Such a test case generation method is seriously blind, resulting in the generation of a large number of invalid test cases, which in turn leads to low code coverage of the test results. That is, according to the evaluation work, the existing RESTful API testing tools cannot generate legal parameter values, resulting in a large number of invalid test requests generated during the test process, which affects the test results and efficiency. Summary of the invention

[0004] In order to solve the technical problem that the existing test request generation method cannot generate legal parameter values, resulting in a large number of invalid test requests being generated during the test process, affecting the test results and efficiency, an embodiment of the present invention provides a method and system for generating API test cases.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a method for generating an API test case, the method comprising: obtaining an invalid test request and its corresponding error response request; obtaining prompt information from the error response request; using the prompt information to modify the error parameters in the invalid test request to generate a new test request.

[0007] In one embodiment, the prompt information includes a status code and a suggestive natural language description, and the use of the prompt information to modify the erroneous parameters in the invalid test request to generate a new test request includes: obtaining a large language model, wherein the large language model can understand the relationship between the status code, the suggestive natural language description and the erroneous parameters in the invalid test request; generating a modification prompt template based on the status code and the suggestive natural language description, and inputting the modification prompt template into the large language model, so that the large language model modifies the erroneous parameters in the invalid test request based on the modification prompt template and outputs a new test request.

[0008] In one embodiment, after generating a new test request, the method further includes: sending the new test request to the target service program for testing to obtain a test result; when the test result is still an error response request, extracting new prompt information from the error response request, and determining whether the new prompt information is the same as the prompt information in the previous error response request; when the new prompt information is different from the prompt information in the previous error response request, continuing to generate a modification prompt template based on the status code and the suggestive natural language description in the prompt information, and inputting the modification prompt template into the large language model, so that the large language model modifies the error parameters in the invalid test request based on the modification prompt template, outputs a new test request, and repeats the above operations until a correct test request is generated.

[0009] In one embodiment, after generating a new test request, the method also includes: determining whether the modification position of the new test request includes a fixed field of the test request endpoint path; if it is determined that the modification position of the new test request includes a fixed field of the test request endpoint path, discarding the new test request; and, determining whether the modification position of the new test request includes an HTTP method in the test request; if it is determined that the modification position of the new test request includes an HTTP method in the test request, discarding the new test request; and, determining whether the modification position of the new test request includes test user identity authentication information, and if it is determined that the modification position of the new test request includes test user identity authentication information, modifying the test user identity authentication information in the new test request to the test user identity authentication information in the original test request.

[0010] In one embodiment, the prompt information is used to modify the error parameters in the invalid test request to generate a new test request, including: after the number of corrections reaches a preset number, if the test result of the generated new test request is still an error response request, constructing a parent request, and using the parent request to access the pre-resource information of the target service program; modifying the error parameters in the invalid test request based on the pre-resource information to generate a new test request.

[0011] In one embodiment, the process of constructing a parent request, sending the parent request to a target service program, and obtaining pre-resource information returned by the target service program includes: constructing a parent request with the same path as the invalid test request, and determining whether the pre-resource information of the target service program can be accessed using the parent request; if it is determined that the pre-resource information of the target service program cannot be accessed using the parent request, recursively reconstructing the parent request in a short path based on the resource dependency implied by the path format, and using the parent request to access the pre-resource information of the upper-layer program of the target service program; and repeating the above operations until the constructed parent request can access the pre-resource information.

[0012] In one embodiment, the constructing of a parent request and using the parent request to access the pre-resource information of the target service program includes: obtaining a suggestive natural language description and a status code in the error response request; generating a prompt template based on the suggestive natural language description and the status code, inputting the prompt template into the large language model, and obtaining a list of error parameter names output by the large language model; constructing a parent request based on the list of error parameter names, and using the parent request to access the pre-resource information of the target service program.

[0013] An embodiment of the present invention also provides an API test case generation system, which includes: an acquisition module for acquiring an invalid test request and its corresponding error response request; a modification module for acquiring prompt information from the error response request; and using the prompt information to modify the error parameters in the invalid test request to generate a new test request.

[0014] An embodiment of the present invention further provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor executes the steps of any of the above methods when running the computer program.

[0015] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0016] The method and system for generating API test cases provided by the embodiment of the present invention obtain an invalid test request and its corresponding error response request; obtain prompt information from the error response request; modify the error parameters in the invalid test request using the prompt information to generate a new test request. The solution provided by the present invention performs targeted corrections on the error parameters in the invalid test request, rather than arbitrarily generating them based on the OpenAPI description document information, which can generate valid test requests with a higher probability and improve the ability to generate valid test cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for generating an API test case according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the framework of the optimized RESTful API fuzzy testing system after adding the RESTX framework in the embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of the structure of the RESTX framework according to an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the execution code of the error parameter correction strategy based on the semantic understanding of the large language model according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of an example of parameter modification in an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of the prompt template format of an embodiment of the present invention;

[0023] Figure 7A schematic diagram of the execution code of the error parameter correction strategy constructed based on the recursive parent request according to an embodiment of the present invention;

[0024] Figure 8 A schematic diagram of the structure of a system for generating API test cases according to an embodiment of the present invention;

[0025] Fig. 9 The figure is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0027] The embodiment of the present invention provides a method for generating an API test case, such as Figure 1 As shown, the method includes:

[0028] Step 101: Obtain an invalid test request and its corresponding error response request;

[0029] Step 102: Obtain prompt information from the error response request; modify the error parameters in the invalid test request using the prompt information to generate a new test request.

[0030] This embodiment utilizes the prompt information in the invalid test request and the error response request received by the test to make targeted corrections to the illegal parameter values ​​in the invalid test request and generate a new test request, which will have a greater probability of generating valid test cases and improve the ability to generate valid test cases.

[0031] The method of this embodiment can make up for the deficiencies of the test case generation process in the prior art, and at the same time, can propose a new parameter value generation strategy for parameters that are difficult to generate legal values ​​in the prior art. The method proposed in this embodiment can be used as an improvement plan of the existing method to improve the effectiveness of the existing RESTful API test case generation method.

[0032] In addition to the process of the traditional RESTful API test case generation method, this embodiment can add a new test case generation stage based on the tested invalid test request. That is, the semantic information in the error response request of the invalid test request and the pre-resources of the target service program are used to correct the error parameters of the invalid test request, thereby generating a new test request. Based on the most classic RESTful API fuzz testing tool RESTler, this embodiment proposes a prototype of a new test case generation method, named RESTX.

[0033] Specifically, see Figure 2 , Figure 2The optimized RESTful API fuzz testing system framework after adding the RESTX framework. In this embodiment, the basic workflow of the optimized part of the system is: during the RESTful API fuzz testing process, when the test case execution and monitoring module receives an error response request with a status code in the 40X range from the target service, the error response request and its corresponding invalid test request are sent to the RESTX module. RESTX will correct the error parameters in the invalid test request according to the prompt information in the dynamic response request, thereby generating a new test request that is more likely to be valid, and then send these new test requests for testing.

[0034] Furthermore, the RESTX module in this embodiment can specifically execute two parameter correction strategies for erroneous parameters, which are an erroneous parameter correction strategy based on semantic understanding of a large speech model and an erroneous parameter correction strategy based on recursive parent request construction.

[0035] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of the RESTX framework. In this RESTX framework:

[0036] Input of the RESTX framework: the error response requests collected during the dynamic fuzz testing process and their corresponding invalid test requests.

[0037] Extract prompt information from error response request: Extract status code and prompt natural language description from dynamic error response request. Here, status code and prompt natural language description are brief summaries of the error cause of invalid test request.

[0038] Two error parameter value correction strategies: RESTX proposes two parameter generation strategies for error parameters, which are based on the semantic information of the error prompt natural language description and the pre-resource information of the test target service program. Using the status code, prompt natural language description and the original invalid request, two strategies are adopted to correct the error parameters in the original invalid request, which are the error parameter correction strategy based on the semantic understanding of the large language model and the error parameter correction strategy based on the recursive parent request.

[0039] Output: Generate new test requests by correcting the original invalid requests. These new test requests will be used for subsequent tests.

[0040] Below, this paper will describe these two parameter correction strategies in detail.

[0041] The first one: Error parameter correction strategy based on semantic understanding of large language model

[0042] The error parameter correction strategy based on the semantic understanding of the large language model utilizes the semantic information of the error suggestive natural language description. The suggestive natural language description extracted from the error response usually contains the format and constraint information of the error parameter. Under the guidance of this information, the correct value can be generated for the error parameter, thereby generating a new test request. Since the Large Language Model (LLM) has excellent semantic understanding capabilities and its familiarity with the HTTP request message format, this embodiment proposes an error parameter correction strategy based on LLM semantic understanding. This strategy uses the LLM model to understand the semantic information in the suggestive natural language description through prompt engineering, and on the basis of semantic understanding, corrects the error parameters in the original invalid request and generates a new test request in the correct format.

[0043] Specifically, in one embodiment, the prompt information includes a status code and a suggestive natural language description, and the use of the prompt information to modify the erroneous parameters in the invalid test request to generate a new test request includes: obtaining a large language model, wherein the large language model can understand the relationship between the status code, the suggestive natural language description and the erroneous parameters in the invalid test request; generating a modification prompt template based on the status code and the suggestive natural language description, and inputting the modification prompt template into the large language model, so that the large language model modifies the erroneous parameters in the invalid test request based on the modification prompt template and outputs a new test request.

[0044] See also Figure 4 , which is a schematic diagram of the execution code of the error parameter correction strategy based on the semantic understanding of the large language model.

[0045] The error parameter correction strategy based on the semantic understanding of the large language model will iteratively correct each invalid test request multiple times until a valid test request is generated for this request template. In each correction iteration, the suggestive natural language description and status code are extracted from the dynamic error response request; a special prompt (i.e., prompt template) is generated based on the invalid test request, status code, and error suggestive natural language description. These prompts are designed to guide the LLM model to modify the incorrect parameters in the invalid test request and generate a new test request.

[0046] The reasons for multiple iterations in this embodiment are as follows: First, the LLM model may not be able to successfully modify the incorrect parameters in the invalid request when it attempts to modify the incorrect parameters for the first time. Second, sometimes the error response request only describes a part of the incorrect parameters. Therefore, the LLM model needs to be repeatedly modified to generate and send new requests, and each test may add additional new prompt information, thereby allowing cumulative revisions.

[0047] Here, it should be noted that if the error response request of the newly generated test request is the same as the error response request of the test request before the LLM model is modified, this strategy will default to the LLM model being unable to modify the error parameters in the invalid request, thereby terminating the modification. In addition, considering that the LLM model may be modified multiple times, the error parameters in the invalid request still cannot be successfully modified. Therefore, this embodiment can set a maximum number of iterations, and the strategy terminates when the maximum number of iterations is exceeded.

[0048] That is, in one embodiment, after generating a new test request, the method further includes: sending the new test request to the target service program for testing to obtain a test result; when the test result is still an error response request, extracting new prompt information from the error response request, and determining whether the new prompt information is the same as the prompt information in the previous error response request; when the new prompt information is different from the prompt information in the previous error response request, continuing to generate a modification prompt template based on the status code and the suggestive natural language description in the prompt information, and inputting the modification prompt template into the large language model, so that the large language model modifies the error parameters in the invalid test request based on the modification prompt template, outputs a new test request, and repeats the above operations until a correct test request is generated.

[0049] The following will be combined Figure 5 , the execution process of the above code is described in detail with a specific embodiment.

[0050] like Figure 5 As shown in a in the figure, the status code of the error response request received by the invalid test request is 400. The prompt natural language description specifically points out that the error parameters are date, date_gmt, menu_order and template, and describes the correct format of the parameters in detail. This strategy uses the prompt natural language description and status code to generate a prompt for modifying the invalid test request. The prompt template format used by this strategy can be as follows Figure 6 As shown. The generated prompt is used to query the LLM model by calling the API, and the LLM model returns the modified new test request.

[0051] Although the LLM model has excellent semantic understanding capabilities, the LLM model occasionally produces incorrect or misleading outputs. Therefore, before the LLM output is regarded as a usable test request, this embodiment can be avoided by strictly verifying it. The verified test request is sent to the target service program for fuzz testing.

[0052] The specific verification rules in this embodiment may be as follows:

[0053] (1) For paths: LLM can only modify the parameter fields within the original request endpoint path. If the fixed fields of the path are modified, that is, the endpoint is modified, the output request no longer belongs to the original request type, so it is unusable.

[0054] This embodiment is based on the typical path design principle of RESTful API. Usually, the path design originally satisfies the regular expression:

[0055] (resourceType / resourceIdentifier / )+

[0056] Take the API path / posts / post_id / revisions / revision_id provided by the typical web application WordPress as an example. The request under this path is to operate the revision object of a specific post, where posts and revisions are two resource types and are fixed fields in the path, while post_id and revision_id are path parameters and can be assigned different values. If LLM mistakenly changes posts and revisions when modifying the request under this path, then the request is no longer to operate the revision object of a specific post, that is, the request no longer performs its original function.

[0057] Specifically, to address this problem, this embodiment establishes the following multiple path checking rules:

[0058] (A) The number of path separators, i.e., ` / `, should be consistent. For example, for a request type under the path / posts / post_id / revisions / revision_id, the original request path is / posts / 1 / revisions / 2. If LLM outputs the modified request path as / posts / 1 / revisions, then this rule can check that the LLM output request path does not match the original request.

[0059] (B) Any two consecutive parts separated by ` / ` should not be modified at the same time. The original request path is / posts / 1 / revisions / 2. If LLM outputs the modified request path as / posts / 1 / comments / 3, this rule can be used to determine that the original request path has been modified by mistake. This check rule is based on the path design principle of RESTful API. Usually, the continuous path part will contain reourceType and resourceIdentifier, so if they are modified at the same time, it means that the fixed field reourceType has been modified.

[0060] (2) Regarding HTTP methods. LLM should not modify the HTTP method of the original request. If the method type is modified, the output will fail validation, so it will be discarded. For example, the method of the original request is PUT / posts / 1 / revisions / 2. If the LLM outputs the modified request path as GET / posts / 1 / revisions / 2, the operation performed on the resource by the modified request has changed and it no longer performs the same function as the original request, so the request cannot pass this validation rule.

[0061] (3) Regarding user authentication information. Modifying the test user's authentication information may affect the results of the test request. Therefore, during verification, the authentication information of the LLM output request is restored to the authentication information of the original test user. For example, if the original request uses the Authorization field in the request header for user authentication, then this rule will check the LLM output request to determine whether it includes this field and whether the field value is the same as the value of the original request. If it is judged as no, the Authorization field of the original request will be added to the LLM output request.

[0062] That is, in one embodiment, after generating a new test request, the method also includes: determining whether the modification position of the new test request includes a fixed field of the test request endpoint path; if it is determined that the modification position of the new test request includes a fixed field of the test request endpoint path, discarding the new test request; and, determining whether the modification position of the new test request includes an HTTP method in the test request; if it is determined that the modification position of the new test request includes an HTTP method in the test request, discarding the new test request; and, determining whether the modification position of the new test request includes test user identity authentication information, and if it is determined that the modification position of the new test request includes test user identity authentication information, modifying the test user identity authentication information in the new test request to the test user identity authentication information in the original test request.

[0063] The above content is related to the error parameter correction strategy based on the semantic understanding of the large language model. The second parameter correction strategy is introduced in detail below.

[0064] Second: Error parameter correction strategy based on recursive parent request construction

[0065] This parameter correction strategy is mainly set for the scenario where some parameters need to rely on the pre-resources of the test target service program. In some scenarios, some parameters need to rely on the pre-resources of the test target service program, and the existing methods are difficult to generate valid parameter values ​​for such parameters when the resource information of the target service program is unknown. Therefore, the existing methods are insufficient in generating legal parameter values. This parameter correction strategy proposes to use the prompt information in the error response request and the pre-resource information of the test target service program to generate new parameter values ​​for these error parameters.

[0066] Specifically, the error parameter correction strategy constructed based on the recursive parent request utilizes the existing API-related resource information of the test target service program to extract legal candidate values ​​for the error parameters. Inspired by the HTTP CRUD semantic design principles and the design characteristics of RESTful API resources, this correction strategy automatically constructs and sends a GET request with the same path as the original invalid request to access the relevant resources. Subsequently, the legal values ​​of the error parameters are extracted from the successful response request. These values ​​are then used to modify the error parameters and generate new test requests for fuzz testing.

[0067] Continue with Figure 5 Take the invalid request in b as an example. The error parameters of the invalid request are author and parent. The values ​​of these two error parameters need to correspond to the identifiers of the existing author and parent resources. Therefore, the HTTP method type of the invalid request is changed to GET, and all parameters in the original request are removed, generating Figure 5 The GET request in c in the above example obtains the resource information corresponding to the invalid request POST wp / v2 / pages, i.e. the page resource list. It can be seen that each page information in the resource list contains legal author and parent values, so the legal value list of these two error parameters can be obtained. Select a legal value and modify the error parameter in the original request to obtain the following: Figure 5 The new test request in d in Received a 201 status code, successfully executed the function corresponding to the request, and created a new page resource object.

[0068] In one embodiment, the prompt information is used to modify the error parameters in the invalid test request to generate a new test request, including: after the number of corrections reaches a preset number, if the test result of the generated new test request is still an error response request, constructing a parent request, and using the parent request to access the pre-resource information of the target service program; modifying the error parameters in the invalid test request based on the pre-resource information to generate a new test request.

[0069] It is worth noting that sometimes the resources related to the current request type may depend on other resources, namely, parent resources, which will cause the GET request to be unable to access any resources. In view of this situation, this embodiment uses the resource dependency relationship implied by the path format to recursively construct a parent request with a short path to access its multi-layer parent resources until the parent resource exists. Then, based on the acquired top-level parent resource, the child resources are gradually acquired until the resources related to the current request type are acquired.

[0070] Take the PUT / posts / 1 / revisions / 2 request as an example. In order to obtain the resource information of this request type operation, a GET / posts / 1 / revisions / 2 request is first directly generated. The resource corresponding to this request (the revision object with id 2 and page object with id 1) does not exist, and the information required by the original invalid request cannot be obtained. This is because the request to obtain the resource contains two request parameters, and the values ​​of these two parameters must be the identifiers of existing resources of the target service (which can be ID, Name, etc.).

[0071] At this point, you need to construct a parent request to obtain the parent resource. The specific method is to gradually delete the path part separated by " / " from the end of the path to construct a request corresponding to the short path. In the current example, the GET / posts / 1 / revisions request is first constructed to obtain a list of all revisions of the page object with id 1.

[0072] If the response to the request is empty, or the request is still an invalid request, the path will continue to be shortened and attempts will be made to obtain the upper-level resource until the resource exists. For example, GET / posts / 1 and GET / posts will be constructed in sequence.

[0073] After GET / posts successfully obtains the top-level parent resource, it will recursively generate long-path requests to obtain resources until the resource information that the original request depends on is obtained. For example, GET / posts obtains the existing page type resource list of the target service. By extracting the id value of the object in the list, it can construct valid requests GET / posts / 23 and GET / posts / 23 / revisions in sequence, and then successfully obtains the list of all revisions resources of the page object with id 23. By extracting the id value of the object in the list, it can successfully construct the GET / posts / 23 / revisions / 45 request, and obtain relevant resource information for the original request.

[0074] Finally, change the path of the original invalid request to PUT / posts / 23 / revisions / 45, that is, only modify the path parameter part, and then change the error parameters in the original request body to the parameter values ​​in the obtained resource information, and you can successfully generate a valid test request.

[0075] That is, in one embodiment, the parent request is constructed, the parent request is sent to the target service program, and the pre-resource information returned by the target service program is obtained, including: constructing a parent request with the same path as the invalid test request, and determining whether the pre-resource information of the target service program can be accessed by using the parent request; when it is determined that the pre-resource information of the target service program cannot be accessed by using the parent request, recursively reconstructing the parent request in a short path according to the resource dependency implied by the path format, and using the parent request to access the pre-resource information of the upper-level program of the target service program; repeating the above operations until the constructed parent request can access the pre-resource information.

[0076] See also Figure 7 , which is a schematic diagram of the execution code of the error parameter correction strategy based on the semantic understanding of the large language model.

[0077] For each invalid test request, this strategy will first extract the suggestive natural language description and status code from the dynamic error response request; then generate a special prompt to guide the LLM model to find the erroneous parameters of the invalid test request and output a list of parameter names; that is, this embodiment will guide the LLM model to find the erroneous parameters through prompt engineering; then, recursively construct a parent request to access the related resources of the invalid request. Extract legal parameter values ​​from the successful response of the request to access the related resources as candidates. After that, this strategy will modify the invalid test request in multiple iterations until a valid test request is generated.

[0078] In each modification iteration, this strategy randomly selects a candidate value for each error parameter to replace its original error value. If the resource does not contain the candidate value of the error parameter, the strategy will delete the error parameter. Then, the new test request will enter the fuzz testing process and be sent to the target service. If the new test request receives a different error response during the test, it will be processed by Algorithm 1, that is, the error parameter correction strategy based on the semantic understanding of the large language model.

[0079] That is, in one embodiment, the construction of the parent request and the use of the parent request to access the prerequisite resource information of the target service program include: obtaining the suggestive natural language description and status code in the error response request; generating a prompt template based on the suggestive natural language description and status code, inputting the prompt template into the large language model, and obtaining the error parameter name list output by the large language model; constructing the parent request based on the error parameter name list, and using the parent request to access the prerequisite resource information of the target service program.

[0080] The test request generation method proposed in this embodiment utilizes the semantic information in the error response request of the invalid test request and the pre-resources of the target service program to perform targeted corrections on the erroneous parameters in the invalid test request, thereby generating a new test request with a higher probability of being valid for testing, thereby improving the effectiveness of generating test cases.

[0081] This embodiment designs two parameter generation strategies that are independent of OpenAPI description document information, implements an error parameter correction strategy based on semantic understanding of a large language model and an error parameter correction strategy based on recursive parent request construction, proposes a compensatory solution for the inability of existing methods to effectively generate parameter values, and improves the ability to generate legal parameter values.

[0082] The method for generating an API test case provided by an embodiment of the present invention obtains an invalid test request and its corresponding error response request; obtains prompt information from the error response request; and uses the prompt information to modify the error parameters in the invalid test request to generate a new test request. The solution provided by the present invention performs targeted corrections on the error parameters in the invalid test request, rather than arbitrarily generating them based on the OpenAPI description document information, and can generate valid test requests with a higher probability, thereby improving the ability to generate valid test cases.

[0083] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a system for generating API test cases, such as Figure 8 As shown, the API test case generation system includes: an acquisition module 801 and a modification module 802; wherein,

[0084] An acquisition module 801 is used to acquire an invalid test request and its corresponding error response request;

[0085] The modification module 802 is used to obtain prompt information from the error response request; use the prompt information to modify the error parameters in the invalid test request to generate a new test request.

[0086] In actual application, the acquisition module 801 and the modification module 802 can be implemented by a processor in the API test case generation system 800.

[0087] It should be noted that: when the above-mentioned device provided in the above-mentioned embodiment is executed, only the division of the above-mentioned program modules is used as an example. In actual application, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the terminal is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the above-mentioned device provided in the above-mentioned embodiment and the above-mentioned method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0088] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0089] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides an electronic device (computer device). Specifically, in one embodiment, the computer device can be a terminal, and its internal structure diagram can be as follows: Fig. 9 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0090] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0100] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0101] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating an API test case, characterized in that: The method comprises: Get invalid test requests and their corresponding error response requests; Acquire prompt information from the error response request; modify the error parameters in the invalid test request using the prompt information to generate a new test request; After generating a new test request, the method further includes: Determining whether the modification position of the new test request includes a fixed field of the test request endpoint path; if it is determined that the modification position of the new test request includes a fixed field of the test request endpoint path, discarding the new test request; and, Determining whether the modified position of the new test request includes the HTTP method in the test request; if it is determined that the modified position of the new test request includes the HTTP method in the test request, discarding the new test request; and, Determine whether the modification position of the new test request includes the test user identity authentication information, and if it is determined that the modification position of the new test request includes the test user identity authentication information, modify the test user identity authentication information in the new test request to the test user identity authentication information in the original test request; The step of modifying the error parameters in the invalid test request by using the prompt information to generate a new test request includes: After the number of corrections reaches the preset number, if the test result of the generated new test request is still an error response request, construct a parent request and use the parent request to access the pre-resource information of the target service program; The erroneous parameters in the invalid test request are modified according to the pre-resource information to generate a new test request.

2. The method for generating an API test case according to claim 1, characterized in that: The prompt information includes a status code and a prompt natural language description, and the using the prompt information to modify the error parameters in the invalid test request to generate a new test request includes: Obtaining a large language model capable of understanding the relationship between the status code, the suggestive natural language description, and the error parameters in the invalid test request; A modification prompt template is generated based on the status code and the suggestive natural language description, and the modification prompt template is input into the large language model, so that the large language model modifies the erroneous parameters in the invalid test request based on the modification prompt template and outputs a new test request.

3. The method for generating an API test case according to claim 2, characterized in that: After generating a new test request, the method further includes: Sending the new test request to the target service program for testing to obtain the test result; If the test result is still an error response request, extract new prompt information from the error response request, and determine whether the new prompt information is the same as the prompt information in the previous error response request; In the case that the new prompt information is different from the prompt information in the previous error response request, continue to generate a modified prompt template based on the status code and the suggestive natural language description in the prompt information, input the modified prompt template into the large language model, so that the large language model modifies the erroneous parameters in the invalid test request based on the modified prompt template, outputs a new test request, and repeats the above operation until a correct test request is generated.

4. The method for generating an API test case according to claim 3, characterized in that: The step of constructing a parent request, sending the parent request to a target service program, and obtaining the pre-resource information returned by the target service program includes: Constructing a parent request with the same path as the invalid test request, and determining whether the parent request can be used to access the pre-resource information of the target service program; When it is determined that the parent request cannot access the prerequisite resource information of the target service program, the parent request is reconstructed in a short path recursive manner according to the resource dependency implied by the path format, and the parent request is used to access the prerequisite resource information of the upper-level program of the target service program; the above operation is repeated until the constructed parent request can access the prerequisite resource information.

5. The method for generating an API test case according to claim 3, characterized in that: The constructing of the parent request and using the parent request to access the pre-resource information of the target service program includes: Obtaining a suggestive natural language description and a status code in the error response request; generating a prompt template based on the suggestive natural language description and the status code, inputting the prompt template into the large language model, and obtaining a list of error parameter names output by the large language model; A parent request is constructed based on the error parameter name list, and the parent request is used to access the pre-resource information of the target service program.

6. A method system for generating API test cases, characterized in that: The test request generating device comprises: An acquisition module, used to acquire invalid test requests and their corresponding error response requests; A modification module, used to obtain prompt information from the error response request; modify the error parameters in the invalid test request using the prompt information to generate a new test request; Wherein, the modification module is also used for: Determining whether the modification position of the new test request includes a fixed field of the test request endpoint path; if it is determined that the modification position of the new test request includes a fixed field of the test request endpoint path, discarding the new test request; and, Determining whether the modified position of the new test request includes the HTTP method in the test request; if it is determined that the modified position of the new test request includes the HTTP method in the test request, discarding the new test request; and, Determine whether the modification position of the new test request includes the test user identity authentication information, and if it is determined that the modification position of the new test request includes the test user identity authentication information, modify the test user identity authentication information in the new test request to the test user identity authentication information in the original test request; Wherein, the modification module is also used for: After the number of corrections reaches the preset number, if the test result of the generated new test request is still an error response request, construct a parent request and use the parent request to access the pre-resource information of the target service program; The erroneous parameters in the invalid test request are modified according to the pre-resource information to generate a new test request.

7. An electronic device, characterized in that: include: A processor and a memory for storing a computer program capable of running on the processor; wherein, When the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 5 are performed.

8. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images