An automated testing method for RPC APIs in large industrial systems

Abstract

This invention discloses an automated testing method for RPC APIs in large-scale industrial systems. The method uses API documentation and tenant historical call data as input to achieve automated API testing. The method mainly comprises three stages: API dependency model construction, API test sequence generation, and API sequence parameter rendering. The API dependency model construction is based on API documentation and tenant historical call data, and consists of three steps: static analysis of the API documentation, parameter clustering based on the large model, and dependency filtering based on tenant historical call data. API test sequence generation is based on the API dependency graph generated in the previous stage. Starting with the API under test, iteratively searching for preceding dependent APIs to generate the sequence, and finally filtering through CRUD semantics. API sequence parameter rendering is achieved by constructing a dynamic dictionary and recommending parameters based on the API sequences generated in the previous stage.

Description

Technical Field

[0001] This invention relates to the field of software testing, and more particularly to an automated testing method for RPC APIs of large industrial systems. Background Technology

[0002] The two mainstream Open API design styles on the market are REST and RPC. Testing REST APIs has been studied since it was first proposed in 2000, but the amount of research has remained limited and the results have been generally poor. However, since the release of EvoMaster and RESTler, research on REST APIs has increased. Research on RPC APIs, however, remains extremely scarce. The following will introduce the corresponding tools related to this invention.

[0003] EvoMaster, a white-box testing tool, accesses the API's source code and uses a genetic algorithm to dynamically analyze Java bytecode, generating test cases. Heuristic functions then guide the generation of test cases to improve code coverage. It was rated as the most effective tool in the review.

[0004] RESTler was the first stateful black-box fuzzing tool for REST APIs. "Stateful" means it can test deeper levels of the tool's state. RESTler analyzes the API documentation to infer producer-consumer dependencies between APIs and, combined with real-time dynamic feedback during test execution, updates the request sequence and records exceptions to complete the test. RESTler and EvoMaster introduced REST API testing to the black-box and white-box testing domains, respectively, and subsequent research has followed.

[0005] Unlike RESTler's modelless random fuzz testing, RestTestGen attempts to address the insufficient modeling of API dependencies by building an ODG (Operation Dependency Graph) to model API dependencies. During testing, calls are made to each leaf node in the ODG, and filtering is performed based on semantics. For data variable matching and numerical rendering, a data name matching fault tolerance mechanism is also implemented to address the issue of inconsistent API documentation quality.

[0006] Compared to RESTler and RestTestGen, which focus on dependencies between multiple APIs, RESTest focuses on dependencies between API parameters, supporting automatic analysis of these dependencies. Parameter dependencies require the use of the Inter-parameter Dependency Language (IDL). IDL needs to be written by developers in advance, and then a constraint solver is used to generate valid test cases.

[0007] Morest combines the strengths of previous works like RESTler and RestTestGen, using a more information-rich RPG (RESTful-service Property Graph) to model dependencies between operations and parameters, and updating the RPG connections during execution to provide feedback updates. The testing process consists of two steps: first, building the RPG using the API documentation, and then dynamically updating the RPG during testing. During testing, multiple test sequences are formed by traversing all parameter nodes and extending them at both ends based on dependency connections.

[0008] RestCT utilizes the t-way technique of combinatorial testing to address the problem of exhaustively exploring the input space, while also considering dependencies and constraints between operations and parameters. The generation process consists of two phases: first, a constraint sequence covering array is generated to determine the order of operation execution; then, an adaptive strategy is applied to concretize several constraint covering arrays. In constraint modeling, RestCT's method is more comprehensive than previous methods: for API dependency modeling, it considers resource hierarchy and CRUD semantics; for parameter constraint modeling, it uses the natural language tool spaCy to design 23 parameter constraint patterns, which are extracted using pattern matching.

[0009] RESTInfer fully leverages the potential of natural language processing, analyzing the description field in API documentation to infer potential parameter constraints. Compared to RESTest, which requires manually writing IDL parameter dependencies, RESTInfer is much more convenient. Its main process is as follows: first, constraint information is extracted using the entity recognition tool NERSE and the relation extraction tool Stanford parser; then, the parameter constraints are formalized into propositional logic; and finally, they are connected to Morest for further experimentation.

[0010] In 2022, Arcuri, the developer of EvoMaster, worked with the Meituan team to apply EvoMaster to the testing of modern RPC APIs. They added a REST layer to the original EvoMaster to adapt it to RPC-style APIs, and also made RPC-compatible modifications to many other parts of EvoMaster, including interfaces, search space, and adapt functions.

[0011] However, existing technologies still have many problems. First, black-box testing tools rely solely on API documentation, which is available for any publicly accessible API; they do not require access to the source code of the object under test, nor do they impose any access requirements or restrictions on the underlying logic and language. White-box testing tools, such as EvoMaster, require access to the source code and runtime process, which involves enterprise information security and poses many risks during the application and use of these tools. Furthermore, they cannot achieve effective testing for many academic researchers or even researchers in non-business teams within enterprises.

[0012] Secondly, RPC APIs lack information compared to REST APIs. Directly adapting existing REST API testing tools to RPC APIs will significantly reduce the effectiveness of testing.

[0013] Tools like RestTestGen and Morest model API dependencies using dependency graphs and generate API sequences by searching the constructed graph. Tools like RestTest rely on manually written IDLs to model constraint dependencies between parameters. Tools like RestCT and RestInfer use template matching and natural language processing, but their methods for analyzing the textual descriptions in API documentation are not accurate or convenient enough. Summary of the Invention

[0014] The purpose of this invention is to address the shortcomings of existing technologies by proposing an automated testing method for RPC APIs in large-scale industrial systems.

[0015] The objective of this invention is achieved through the following technical solution: an automated testing method for RPC APIs of large-scale industrial systems, the method comprising:

[0016] S1. Perform static analysis on RPC-style API documentation to obtain API parameter information and API dependency graph;

[0017] S2. Based on the large model, the similarity of API parameters is re-evaluated, and new dependencies are added to the API dependency graph;

[0018] S3. Filter the API dependency graph based on tenant historical call data to obtain a more accurate and usable API dependency graph;

[0019] S4. Automatically generate API call sequence: First, the edges of the API dependency graph are weighted according to the tenant's historical call data. Then, the API sequence is generated based on the weighted API dependency graph. Finally, the API sequence is filtered according to the API's CRUD attributes extracted from the large model to obtain an API sequence that can be used for testing.

[0020] S5. Automatically fill parameters for API sequences to form complete API test cases: Use the dynamic parameters generated by the API sequence to fill in some parameters for subsequent APIs, and automatically fill in the remaining parameters based on the large language model. The input of the large model includes parameter-related information extracted from the API documentation, and the output of the large model is a recommended combination of API parameters.

[0021] S6. The complete test cases will be encoded and encrypted according to the request specifications of the test server, and then tested on the test server to obtain the final test results.

[0022] Furthermore, the static analysis specifically includes: parsing the JSON-formatted API document, extracting the API name, API meaning, API parameters and their descriptions, API return information, error codes and error messages, and storing them in the API parameter information for subsequent analysis.

[0023] Furthermore, S2 specifically includes:

[0024] Add API dependencies based on parameter similarity; extract the input and output parameters of the API, first determine the correlation between parameters based on the strong correlation between them, cluster and connect related output and input parameters, add new dependencies, with the direction pointing from the API corresponding to the output parameter to the API corresponding to the input parameter.

[0025] Then, a prompt is designed to determine whether there is a correlation between different parameters through a large model, to measure the weak correlation between parameters, and to add new dependencies in the same way.

[0026] In the determination of the relevant items, if the number of parameters is small, the pairwise comparison between parameters is adopted for judgment. If the number of parameters is large, a two-step method of coarse screening + fine screening is adopted. First, all parameters and their brief descriptions are combined into prompt words and preliminary clustering is performed through a large model. Then, in each coarse screening group, more structural information is provided for the parameters, and a second clustering is performed to obtain the final classification result. Parameters in the same class are considered as relevant parameters.

[0027] Furthermore, the filtering of API dependency graph dependencies based on tenant historical call data specifically includes: collecting historical call data of different users within a certain period of time, inferring whether there may be dependencies between APIs by limiting the API call time interval and whether APIs have input and output parameters with the same parameter values; verifying the obtained API dependencies, and if there is actual call data to prove it, then the API dependency is considered to exist, otherwise it is deleted.

[0028] Furthermore, the weighting of the edges in the API dependency graph based on tenant historical call data specifically involves assigning a weight to each dependency relationship as the frequency of that API dependency relationship in the historical data. The formula is shown below:

[0029]

[0030] Furthermore, the step of generating API sequences based on the weighted API dependency graph includes: starting with a specified API to be tested, performing an iterative search on its dependencies to obtain the API request sequence for each API to be tested; if some input parameters cannot find corresponding preceding APIs to generate, they are considered as parameters that need to be specified separately, and will be specified by the user or recommended by the large model later. During the search process, the access order will be recorded and loops will be ignored to prevent entering an infinite loop.

[0031] Furthermore, the specific formula used in S4 to filter the API sequence based on the CRUD attributes of the APIs extracted from the large model is shown below:

[0032]

[0033] Furthermore, the prompts for the large model in the large model recommendation parameters include: role description, API documentation, output format requirements, and special output restrictions;

[0034] The role description informs the large model that it will play the role of a test development engineer, thus providing the task background; the API documentation provides relevant API information; the output format requires that the returned data be a JSON file that the tool can directly use; the output special restrictions are used to prevent unexpected behavior; finally, the data returned by the large model is processed by a JSON format analyzer, and responses that do not meet the requirements are marked and re-requested until the format meets the requirements; the filtered information will be used as a parameter recommendation combination to automatically populate API parameters.

[0035] On the other hand, this invention also provides an automated testing device for RPC APIs of large industrial systems, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the automated testing method for RPC APIs of large industrial systems.

[0036] On the other hand, this specification also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned automated testing method for RPC APIs of large industrial systems.

[0037] The beneficial effects of this invention are:

[0038] This invention pertains to black-box testing tools. By automatically analyzing API documentation and generating parameters and sequences for automatic testing, it achieves automated API testing and reduces reliance on manual intervention.

[0039] This invention constructs an API dependency graph based on parameter similarity and corrects it using tenant call records. Given the lack of rich information such as hierarchical structure in RPC APIs, it enhances the understanding of API parameters through a large model, improving the accuracy of the automatically generated API dependency graph and thus increasing test sequence coverage and effectiveness. This invention also utilizes actual production data, introducing its inherent dependencies, effectively reducing the impact of errors caused by API documentation mistakes, improving error detection rate, and enhancing the reliability of test cases.

[0040] This invention uses dynamic parameters and large-scale model-recommended parameters to populate parameter values. Compared to random or sample parameters, dynamic parameters improve the accuracy of parameter information; compared to tools using other natural language processing techniques, this invention uses a large-scale model to improve the understanding of API parameter information; at the same time, it provides more direct parameter combination recommendations, reducing process complexity and improving testing efficiency. Attached Figure Description

[0041] Figure 1 A system architecture diagram provided for embodiments of the present invention;

[0042] Figure 2 This is an example of API documentation provided for embodiments of the present invention;

[0043] Figure 3 Examples of prompt words for two-step clustering provided in embodiments of the present invention;

[0044] Figure 4 This is an example of an API dependency graph provided in an embodiment of the present invention;

[0045] Figure 5This is a schematic diagram of an automated black-box testing device provided in an embodiment of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, this invention provides an automated testing method for RPC APIs in large-scale industrial systems. The design of this invention includes API test sequence generation and API sequence parameter rendering. API sequence generation includes constructing an API dependency model and generating API sequences based on that dependency model. API parameter rendering, on the other hand, specifies the parameters of the previously generated API sequences.

[0048] Specifically, this invention first constructs an API dependency model. The first step involves statically analyzing RPC-style API documentation to obtain API parameter information and an API dependency graph. The second step involves re-evaluating parameter similarity based on a larger model, adding new dependencies to the API dependency graph. The third step involves filtering the dependencies in the API dependency graph based on tenant historical call data. Finally, a more accurate and usable API dependency graph is obtained.

[0049] Then, this invention automatically generates API call sequences. First, the edges of the API dependency graph are weighted based on the tenant's historical call data. Then, an API sequence is generated based on the weighted API dependency graph. Finally, the API sequence is filtered based on the CRUD attributes of the APIs extracted from the large model to obtain an API sequence that can be used for testing.

[0050] Finally, this invention automatically fills in parameters for API sequences to form complete API test cases. On one hand, this invention utilizes dynamic parameters generated from the API sequence to fill in some parameters for subsequent APIs; on the other hand, it automatically fills in the remaining parameters based on a large language model. The input to the large model includes parameter-related information extracted from the API documentation, and the output of the large model is a recommended combination of API parameters.

[0051] Therefore, this invention enables automated black-box testing of RPC APIs in large-scale industrial systems.

[0052] This method can be applied to various RPC-style API testing scenarios, including but not limited to: API testing for Alibaba Cloud's compute, storage, network, and CDN services; API testing for Tencent Cloud's cloud servers, cloud disks, and cloud functions; and API testing for AWS's EC2, VPC, and RDS services. First, the tool will read the Open API documentation. An example of the API documentation is shown below. Figure 2As shown, it generally includes information such as API name, API description, API type, API input parameters and specific specifications, API output parameters and specific structure, possible error codes and their corresponding meanings.

[0053] Specifically, the first part of this invention is based on the construction of an API dependency model based on parameter similarity and historical call data.

[0054] The first step is to perform static analysis. The JSON-formatted API documentation is parsed to extract information such as the API name, API meaning, API parameters and their descriptions (including but not limited to name, type, default value, example value, and meaning), API return information (i.e., return parameters and their descriptions), error codes, and error messages. This information is stored in the API parameter information for subsequent analysis.

[0055] The second step is to add API dependencies based on parameter similarity. Extract the input and output parameters of the API. First, determine the correlation between parameters based on strong correlations (whether parameter names are the same, whether parameter specifications are consistent). Cluster and connect related input and output parameters, adding new dependencies, with the direction pointing from the API corresponding to the output parameter to the API corresponding to the input parameter. Then, design a prompt to determine whether different parameters are correlated through a large model, measuring weak correlations and adding new dependencies using the same method. Figure 3 The image shows an example of a two-step clustering prompt provided in an embodiment of the present invention. It includes three main parts: task description, parameter information, and output format. The first part describes the role and corresponding capabilities of the large model, and describes the task to be performed. For example... Figure 3 The two templates define the large model as a cloud service expert, capable of solving cloud infrastructure-related problems. The second part describes the required parameter information. In the coarse screening stage, the parameter information is a list storing the names of all parameters; in the fine screening stage, it includes the names of parameters within each group and summarized descriptions, provided as key-value pairs. The third part explains the expected output and its format. Both stages are grouping tasks, so the large model is expected to group parameters, with parameters within each group having similar meanings. Regarding the format, the output should return multiple parameter groups, including group numbers and the corresponding parameters within each group, with a JSON example provided.

[0056] For determining the relevance between parameters, this invention designs two different strategies based on the number of parameters. For a small number of parameters, pairwise comparisons are used to obtain better matching results through richer information. For a large number of parameters, a two-step method of coarse screening followed by fine screening is employed. First, all parameters and their brief descriptions are grouped into prompt words, which are then preliminarily clustered using a large model. Then, within each coarse screening group, more structural information is provided to the parameters, and a second clustering is performed to obtain the final classification result. Parameters within the same cluster are considered relevant. The number of parameters depends primarily on the context window size of the large model and its ability to handle long texts. If the number of parameters is large, and the prompt word length of the pairwise comparison scheme exceeds the context window length of the large model, or if the length is too long and the large model's processing performance is poor, then the second scenario applies, and the two-step approach should be used.

[0057] The third step is to filter API dependencies based on tenant historical call data. Historical call data from different users within a selected time period is collected. By limiting the API call time interval and whether APIs have input / output parameters with the same values, potential dependencies between APIs are inferred. The API dependencies obtained in the first two steps are verified. If actual call data supports this, the API dependency is considered to exist; otherwise, it is deleted. The selected time period is determined based on the time range obtained by the tool user.

[0058] This ultimately forms an API dependency graph, an example of which is shown below. Figure 4 As shown, blue nodes represent an API, red nodes represent parameters, and directed edges between nodes represent the relationship between the API and the parameter. A directed edge from a blue node to a red node indicates that the parameter is one of the output parameters of the API, while a directed edge from a red node to a blue node indicates that the parameter is one of the input parameters of the API.

[0059] The second part is the generation of API sequences based on a weighted dependency graph.

[0060] First, the API dependency graph is weighted based on tenant historical call data. The weight of each dependency is the frequency of that API dependency in the historical data. The formula is as follows:

[0061]

[0062] Where weight refers to the weight of the dependency in the directed graph; API begin In a directed graph, APIend refers to the starting API of a dependency relationship, i.e., the API that is depended upon; APIend refers to the ending API of a dependency relationship, i.e., the API that depends on the preceding API; PARAM iIn a directed graph, the dependency relationship formed through the i-th parameter refers to the dependency relationship between two specified APIs achieved through this parameter; N i This refers to the number of times a dependency relationship is established through the i-th parameter after specifying the start and end APIs by analyzing the tenant's historical call data. The condition refers to the judgment criteria we designed for the tenant's historical call data. Specifically: if within a certain time interval (e.g., 5 minutes), the same user accesses two APIs sequentially, and a certain output parameter value of the first accessed API is equal to a certain input parameter value of the second accessed API, then these two APIs are considered to have established a dependency relationship through this parameter value. If the parameter names are equal, the parameter name can be directly used to refer to PARAM. i Otherwise, use parameter pairs (an output parameter of the previous API and an input parameter of the next API).

[0063] Then, sequence generation is performed, starting with the specified API under test and iteratively searching its dependencies. This involves continuously searching forward until all preceding APIs are covered, thus obtaining the API request sequence for each API under test. If certain input parameters cannot find a corresponding preceding API for generation, they are considered parameters that need to be specified separately and will be specified by the user or recommended by the larger model later. Since cycles may appear in the graph, the access order is recorded during the search process, and cycles are ignored to prevent entering an infinite loop.

[0064] Finally, these API dependency sequences are sorted to determine their testing order. The score for an API dependency sequence is calculated by summing the weights of each dependency in the sequence, corresponding to the first term (α) of the following formula. Many other metrics are also included: (a) API sequence length: Shorter sequences are more likely to be used in real-world scenarios because developers aim to simplify API usage. On the other hand, longer sequences may introduce redundant APIs. This corresponds to the second term (β) of the following formula. (b) Number of manual inputs: Fewer manual inputs indicate less human intervention and stronger automation. This corresponds to the third term (γ) of the following formula. (c) CRUD semantics: This will be discussed in the next paragraph. By utilizing these criteria, we can effectively sort the API request sequence for each API under test using the following equation, corresponding to the δ term of the following formula.

[0065] The calculation formula used is as follows:

[0066]

[0067] Where "score" refers to the score of the API sequence; API_Seq{a1,a2,…,a…} n} refers to a iThe API sequence refers to the order of the APIs; α is the coefficient of the first term, the sum of the API sequence weights, which measures the importance of the API sequence by summing the weights of the dependencies in each API sequence; β is the coefficient of the second term, the number of APIs in the API sequence, where n is the number of APIs in the API sequence; γ is the coefficient of the third term, the number of parameters requiring manual input in the API sequence, where N is the number of APIs in the API sequence. Manual_Input This refers to the sum of the number of parameters that require manual input in the API sequence; δ is the coefficient of the API sequence score, which is measured by whether the API sequence conforms to CRUD semantics. That is, it examines any two APIs before and after each other, if they operate on the same resource, whether they conform to the CRUD semantic order of first create - then read and update - finally delete.

[0068] The third part is the generation of API parameters based on a large model.

[0069] API parameter generation is achieved by combining dynamic parameters output by the API with parameters recommended by the large model.

[0070] On one hand, based on the API dependency graph, some parameters in the API sequence can be filled from the output of the previous API into the input of the next API. These parameters are called dynamic parameters because they are dynamically generated by the preceding API during testing and used by the subsequent API. They are stored in a dynamic parameter dictionary for automatic parameter filling.

[0071] On the other hand, API documentation will be provided to the large model to recommend parameter combinations, supplementing the dynamic parameter dictionary and enabling automatic filling of remaining parameters. The prompts for the large model include role descriptions, API documentation, output format requirements, and special output restrictions. The role description informs the large model that it will play the role of a test development engineer, thus providing the task context; the API documentation provides relevant API information, giving the large model a source of information; the output format requirement is to return a JSON file that the tool can directly use; special output restrictions are set to prevent excessive output or unexpected behavior such as outputting illegal characters, as the large model's understanding of the task may easily deviate. Finally, the data returned by the large model is processed by the JSON format analyzer of this invention to verify whether the string returned by the large model can be successfully serialized into a Python dictionary, and to verify whether it contains key-value pairs of parameter combination names, parameter names, and parameter values. If the verification is successful, it can be saved for use by the testing tool. If an answer that does not meet the requirements is identified, it is marked as an error, and a re-request is made until the format meets the requirements. The filtered information will be used as a parameter recommendation combination to automatically fill in the API parameters.

[0072] Finally, test case generation and execution are performed. For API sequences generated based on the API weighted dependency graph, the API parameters are populated using a dynamic parameter dictionary and parameters recommended by a large model to form the original test cases. These test cases are encoded according to the Alibaba Cloud API request specifications, encrypted using the account AK / SK, and sent to Alibaba Cloud servers for test requests.

[0073] After the test is completed, the tool will provide a test report in log format, which includes rich information such as the parameters and responses of each request, and finally provides the API coverage and test case effectiveness. The experience data generated during the test will also be stored in JSON format for subsequent testing, including but not limited to the parameter combinations of actual requests and corresponding response information, API dependency graph data, API parameter information, etc.

[0074] The RPC-style API in this invention is only for differentiation from the REST-style API. For API designs that do not adhere to these two design styles, if the APIs do not have a clear hierarchical relationship, they can be decoupled and independent, and relevant information about input and output parameters can be provided. The tools of this invention can also be used for testing.

[0075] Although this invention is geared towards API testing, it can also be used to test interfaces or similar services that do not expose their internal implementation logic but only provide input and output information, through simple adaptation (such as writing equivalent API documentation).

[0076] Corresponding to the aforementioned embodiment of an automated testing method for RPC APIs of large industrial systems, the present invention also provides an embodiment of an automated testing device for RPC APIs of large industrial systems.

[0077] See Figure 5 The present invention provides an automated testing device for RPC APIs of large industrial systems, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements an automated testing method for RPC APIs of large industrial systems as described in the above embodiment.

[0078] The embodiment of the automated testing device for RPC APIs of large-scale industrial systems provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, used in an automated testing device for RPC APIs of large industrial systems provided by this invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0079] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0081] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements an automated testing method for RPC APIs of large industrial systems as described in the above embodiments.

[0082] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0083] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned automated testing method for RPC APIs of large industrial systems.

[0084] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0085] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An automated testing method for RPC APIs in large-scale industrial systems, characterized in that, The method includes: S1. Perform static analysis on RPC-style API documentation to obtain API parameter information and API dependency graph; the static analysis specifically includes: parsing the JSON-formatted API documentation, extracting the API name, API meaning, API parameters and their descriptions, API return information, error codes and error messages, and storing them in the API parameter information for subsequent analysis; S2. Based on the large model, re-evaluate the similarity of API parameters and add new dependencies to the API dependency graph, including: adding API dependencies based on parameter similarity; extracting the input and output parameters of the API, first judging the correlation of parameters based on the strong correlation between parameters, clustering and connecting related input and output parameters, adding new dependencies, with the direction from the API corresponding to the output parameter to the API corresponding to the input parameter. Then, prompt words are designed, and the large model is used to determine whether there is a correlation between different parameters. The weak correlation between parameters is measured, and the relevant output and input parameters are clustered and connected to add new dependencies. S3. Filter the API dependency graph based on tenant historical call data to obtain a more accurate and usable API dependency graph; S4. Automatically generate API call sequence: First, the edges of the API dependency graph are weighted according to the tenant's historical call data. Then, the API sequence is generated based on the weighted API dependency graph. Finally, the API sequence is filtered based on the CRUD attributes of the API extracted from the large model to obtain an API sequence that can be used for testing. S5. Automatically fill parameters for the API sequence to form complete API test cases: Use the dynamic parameters generated by the API sequence to fill in some parameters for subsequent APIs, and automatically fill in the remaining parameters based on the large model. The input of the large model includes parameter-related information extracted from the API documentation, and the output of the large model is a recommended combination of API parameters; the dynamic parameters are the parameters obtained by filling the output parameters of the previous API into the input parameters of the next API. S6. Encode and encrypt the complete test cases according to the request specifications of the test server, and perform the test on the test server to obtain the final test results.

2. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, S2 specifically includes: In the determination of relevance, different strategies are used based on the context window size of the large model and its ability to process long texts. If the number of parameters does not affect the effect of pairwise comparison schemes, then pairwise comparison between parameters is adopted for judgment. If the number of parameters is large, causing the length of the prompt words in the pairwise comparison scheme to be greater than the context window length of the large model, or if the length is too long and the large model's processing effect is poor, then a two-step method of coarse screening + fine screening is adopted. First, all parameters and their brief descriptions are combined into prompt words, and the large model performs preliminary clustering. Then, in each coarse screening group, more structural information is provided for the parameters, and a second clustering is performed to obtain the final classification result. Parameters in the same class are considered as related parameters.

3. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, The filtering of API dependency graph based on tenant historical call data specifically includes: collecting historical call data of different tenants within a selected time period; inferring whether there is a dependency relationship between APIs by limiting the API call time interval and whether the APIs have the same input and output parameter values; verifying the obtained API dependency relationship; if there is actual call data to prove it, the API dependency relationship is considered to exist, otherwise it is deleted.

4. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, The weighting of the edges in the API dependency graph based on tenant historical call data specifically involves the following: the weight of each dependency relationship is the frequency of occurrence of that API dependency relationship in the historical data, as shown in the formula below: ， Where weight refers to the weight of the dependency in the directed graph; API begin In a directed graph, APIend refers to the starting API of a dependency relationship, i.e., the API that is depended upon; APIend refers to the ending API of a dependency relationship, i.e., the API that depends on the preceding API; PARAM i In a directed graph, the dependency relationship formed through the i-th parameter refers to the dependency relationship between two specified APIs achieved through this parameter; N i This refers to the number of times the dependency relationship implemented through the i-th parameter occurs after specifying the start and end APIs by analyzing the tenant's historical call data; "condition" refers to the judgment condition we designed for the tenant's historical call data. If the parameter names are equal, the parameter name is used directly to refer to PARAM. i Otherwise, use parameter pairs.

5. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, The process of generating API sequences based on a weighted API dependency graph includes: starting with a specified API to be tested, iteratively searching its dependencies to obtain the API request sequence for each API to be tested; if some input parameters cannot find corresponding preceding APIs to generate, they are considered as parameters that need to be specified separately, and will be specified by the user or recommended by the large model later. During the search process, the access order will be recorded and loops will be ignored to prevent entering an infinite loop.

6. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, The specific formula used in S4 to filter the API sequence based on the CRUD attributes of the API extracted from the large model is shown below: ， Where "score" refers to the score of the API sequence; API_Seq{a1, a2,…, a n } refers to a i The API sequence refers to the order of APIs; α is the coefficient of the first API sequence weight sum, which measures the importance of the API sequence and is achieved by summing the weights of the dependencies of each API sequence. β is the coefficient of the number of APIs in the second API sequence, and n is the number of APIs in the API sequence; γ is the coefficient of the number of parameters that require manual input in the third API sequence, N Manual_Input This refers to the sum of the number of parameters that require manual input in the API sequence; δ is the coefficient of the API sequence score, which is measured by whether the API sequence conforms to CRUD semantics. That is, it examines any two APIs before and after each other, if they operate on the same resource, whether they conform to the CRUD semantic order of first creating, then reading and updating, and finally deleting.

7. The automated testing method for RPC APIs of large-scale industrial systems according to claim 1, characterized in that, In the process of automatically filling in the remaining parameters based on the large model, the prompts given to the large model include: role description, API documentation, output format requirements, and special output restrictions; The role description informs the large model that it will act as a test development engineer, thus providing the task background; the API documentation provides relevant API information; the output format requires a JSON file that the tool can directly use; special output restrictions are used to prevent unexpected behavior; finally, the data returned by the large model is processed by a JSON format analyzer, which marks answers that do not meet the requirements and re-requests until the format meets the requirements; the filtered information will be used as a parameter recommendation combination to automatically fill in API parameters; the JSON format analyzer verifies whether the string returned by the large model can be successfully serialized into a Python dictionary and whether it contains key-value pairs of parameter combination names, parameter names, and parameter values; if the verification is successful, it is saved; if an answer that does not meet the requirements is identified, it is marked as an error.

8. An automated testing device for RPC APIs of large industrial systems, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an automated testing method for RPC APIs of large industrial systems as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an automated testing method for RPC APIs of large industrial systems as described in any one of claims 1-7.

Images