Test case generation method and system based on dependency graph

Abstract

The invention belongs to the field of software testing, and particularly relates to a test case generation method and system based on a dependency graph, and the method comprises the steps: constructing a self-adaptive dependency graph model OPG = (V, E, mu); generating a test sub-graph by adopting an OPG traversal algorithm and a reverse order generation algorithm; and generating test parameters in real time in the test process by adopting a method of combining multiple generation strategies. The dependency relationship graph model constructed in the method can better reflect the dependency relationship between operations, the quality of the generated test cases is improved from the two aspects of operation and parameters, and based on a good feedback mechanism, the number of error parameters can be reduced, the number of the generated test cases is further reduced, and the problem that the scale of the test cases is too large is avoided; and meanwhile, the test sub-graph is generated as a test case to reflect a more complex logic relationship between operations, so that the effectiveness of the test case can be improved, and the coverage rate of a test target is further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a test case generation method and system based on a dependency graph. Background Art

[0002] With the development of related technologies, the scale of network services has grown rapidly, among which cloud services used to build distributed services and data processing, provide basic computing measures and software hosting services have exploded. REST (Representational State Transfer) is a software architecture first proposed by Roy Fielding in 2000. It standardizes the operations of adding, deleting, modifying and checking (CRUD) resources, and aims to provide a simple, lightweight, scalable and reliable way to build distributed systems and network services. RESTful API is an application interface design method that conforms to the REST architectural style. The design concept of RESTful API is simple and lightweight. It relies on the HTTP protocol for data interaction. It has the advantages of cross-platform and cross-language, and is suitable for various distributed systems and microservice architectures. Based on the above characteristics, RESTful API has become part of the standardized interface in cloud computing platforms (such as AWS and Azure), providing users with convenient cloud management tools. Among them, OpenAPI has become the mainstream RESTful API interface description language. Cloud services that follow the OpenAPI specification will provide users with a specification document that describes the endpoints and parameters in the cloud service, making it convenient for users to access the cloud service through requests.

[0003] In RESTful API fuzz testing, the OpenAPI specification document is usually analyzed to obtain the dependency relationship between operations. The dependency relationship here refers to the consumer-producer relationship proposed by the Microsoft team. The consumer needs a certain parameter value when executing, and the producer can obtain a certain parameter after execution. This requirement relationship constitutes a dependency relationship. After analyzing the dependency relationship, an operation sequence consisting of multiple operations is generated as a stateful test case based on the dependency relationship. For example, create, access, and delete constitute a test sequence as a test case. The purpose of generating stateful test cases is to test the operation nodes that cannot be directly tested through a single node through a long operation sequence, improve the test coverage, and expand the test scope of the service.

[0004] Test case generation strategy refers to the automatic generation of test cases based on a series of algorithms or techniques. Currently, the common RESTful API test case generation strategies in black box testing are: model-based test case generation strategy and random test case generation strategy. Among them, the model-based test case generation strategy abstracts the program into a mathematical model, abstracts the system state into a state diagram or state transition matrix, and then applies a search algorithm to search in the state space to generate as many and representative test cases as possible. Random testing is to construct test cases by assigning random values ​​to each parameter of the operation under test. Fuzz testing is a widely used randomized testing method in RESTful API testing. Its core idea is to input automatically or semi-automatically generated random data into the program and monitor program exceptions, such as crashes or assertion failures, to find possible program errors.

[0005] In 2019, the Microsoft team proposed the Restler (Atlidakis V, Godefroid P, Polishchuk M. Restler: Stateful Rest API Fuzzing. IEEE / ACM the 41st International Conference on Software Engineering (ICSE). 748-758, 2019.) black-box fuzz testing tool. After analyzing the OpenAPI specification document (Swagger. API Development for Everyone. http: / / swagger.io / .2020.), a grammar file that records dependency and parameter information was obtained. According to the grammar file, a breadth-first or randomly selected search strategy was used to generate a stateful test sequence for testing. However, the search strategy used by Restler has the problem of path explosion, resulting in insufficient length and low quality of the generated sequence. At the same time, there is a problem of dynamic feedback of test parameters, resulting in mediocre test results.

[0006] In 2020, Emanuele Viglianisi et al. from Italy proposed RestTestGen (Viglianisi E, Dallago M, Ceccato M. Resttestgen: Automated Black-box Testing of RESTful APIs. 2020 IEEE the 13th International Conference on Software Testing, Validation and Verification (ICST). 142-152, 2020.), which uses a test case generation method based on an operation dependency graph, and proposes a dependency graph model composed of operations as graph nodes and dependencies as edges. The graph model is used to record the dependencies analyzed from the document and test based on the graph. It does not rely on the breadth-first search strategy, but generates operation sequences from top to bottom. However, since there are only operation nodes in the dependency graph model designed, the graph structure is too thin, resulting in a large number of identical test cases being repeatedly generated during the testing process. In addition, the construction of the graph model is too dependent on the API specification document. When there are some problems with the specification document, the dependency graph cannot correctly reflect the dependencies between operations, which in turn affects the quality of the generated test cases.

[0007] In 2022, Huawei's team proposed the Morest (Liu Y, Li Y, Deng G, et al. Morest: Model-based RESTful API Testing with Execution Feedback. Proceedings of the 44th International Conference on Software Engineering. 1406-1417, 2022.) testing tool, which uses a test case generation method based on a service attribute graph. On the basis of the operation dependency graph, a service attribute graph is further proposed. The service attribute graph is a directed, edge-labeled, attributed multi-graph model that records the relationship between operations and the relationship between resources. Based on the service attribute graph, the Cartesian product calculation method is used to traverse the resource nodes in the graph to generate a test sequence, and a dynamic feedback mechanism is added to make up for the shortcomings of the first two. However, the test case generation algorithm and the characteristics of the resource nodes lead to a certain degree of randomness between sequences, low logic, low quality of test cases, and few resource nodes in the service attribute graph, resulting in a small number of test cases generated and low test coverage. Summary of the invention

[0008] At present, when different methods are used to generate stateful test cases during RESTful API fuzz testing, the inaccuracy and incompleteness of the analysis and recording of the dependencies between operations result in low effectiveness of the generated test cases, which directly affects the endpoint coverage of the test and the resources consumed by the test. Therefore, in order to solve the problems of low effectiveness, excessive scale and insufficient coverage of the test cases generated by the existing methods during the RestfulAPI fuzz testing process, the present invention provides a test case generation method and system based on a dependency graph. The present invention constructs a dependency graph model composed of parameters and operations, uses the characteristics of the graph to reflect and utilize the dependencies between operations, and updates the dependency graph model in real time according to feedback during the process to optimize the effectiveness of the generated test cases. This can effectively improve the effectiveness of the test cases and reduce the scale of the test cases, thereby optimizing the fuzzy testing effect.

[0009] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0010] The present invention provides a test case generation method based on a dependency graph, comprising the following steps:

[0011] Step S1: construct an adaptive dependency graph model;

[0012] The dependency graph model is: OPG = (V, E, μ); V represents a node set, V = V parameter ∪V operation , V operation Represents the set of operation nodes, V parameter represents the parameter node set; E represents the directed edge set, E=E op ∪E po , E op represents the set of edges from the operation to the parameter, E po represents the edge set from parameter to operation; μ represents the mapping function set between operation and parameter to add attribute value, μ = μ parameter ∪μ operation , μ parameter Represents a function that adds attributes to a parameter, μ operation A function that represents the operation of adding attributes;

[0013] Step S2: Establish a stateful test case generation algorithm;

[0014] S2.1: Generate test subgraphs using OPG traversal algorithm and reverse generation algorithm;

[0015] S2.2: Use a combination of multiple generation strategies to generate test parameters in real time during the test process.

[0016] Furthermore, in step S2.1, a test subgraph and sequence are initialized, the node a to be tested is set as the root node of the test subgraph, and is added to the sequence.

[0017] Furthermore, in step S2.1, the parent node of node a is accessed in the dependency graph model to obtain the parameter value required for the operation, and the parent node of each parameter value is accessed to obtain the operation set that can produce the parameter.

[0018] Furthermore, in step S2.1, operations that satisfy the producer-consumer relationship, CRUD semantic constraints and do not form a loop are selected, and parameter nodes and operation nodes are added to the test subgraph in turn. The operation nodes are also appended to the front of the sequence in turn, and the above operations are repeated until all operation nodes in the sequence are visited.

[0019] Furthermore, in step S2.2, the generation strategies include: a success value generation strategy, a response value generation strategy, an example value / default value generation strategy and a random value generation strategy.

[0020] Furthermore, the generation strategies are arranged in order according to the optimization level: success value generation strategy, response value generation strategy, example value / default value generation strategy, and random value generation strategy.

[0021] Furthermore, in step S2.2, a success value generation strategy is first selected. If a success value does not exist, it is obtained from the response value. These generation strategies are applied in descending order according to the optimization level until the value range of each test parameter is determined.

[0022] Furthermore, the success value is the value saved in the previous successful test case; the response value is the value analyzed from the test feedback message, which is divided into a real-time response value and a total response value; the example value / default value is the value analyzed from the RestfulAPI specification document; the random value is: when the above success value, response value, example value / default value cannot obtain a valid data value, the data is randomly generated according to the parameter requirements.

[0023] The present invention provides a test case generation system based on a dependency graph, which is used to implement the test case generation method based on a dependency graph. The system includes: an adaptive dependency graph model construction module, a test subgraph generation module and a test parameter generation module; the adaptive dependency graph model construction module is used to construct an adaptive dependency graph model; the test subgraph generation module includes an OPG traversal algorithm module and a reverse generation algorithm module, the OPG traversal algorithm module is used to traverse each operation node in the dependency graph model, and the reverse generation algorithm module is used to generate a test subgraph according to the operation node; the test parameter generation module is used to adopt a method combining multiple generation strategies to generate test parameters in real time during the test process.

[0024] The present invention provides an electronic device, comprising:

[0025] Memory, used to store computer programs and local data uploaded by users;

[0026] A processor, configured to execute a computer program stored in a memory to implement the steps of the test case generation method based on a dependency graph;

[0027] Communication interface, used for communication between electronic equipment and other devices;

[0028] The communication bus is used to realize communication between the processor, the communication interface and the memory.

[0029] The beneficial effects of the present invention are:

[0030] The present invention provides a test case generation method and system based on a dependency graph. By analyzing a RESTful API specification document, a dependency graph model composed of parameters and operations is constructed. Parameters and operations are used as nodes in the graph structure, which can more effectively reflect the logical relationship between parameters and operations, and realize real-time update of parameter data and edge relationships during the test process. At the same time, the problem of too few generated test cases and low coverage caused by too few nodes in the graph model of Morest can be avoided. At the same time, the present invention adopts a test subgraph generated by a reverse generation algorithm as a test case. As a test case, the test subgraph can test more complex logical relationships instead of a single linear logical relationship, and is more suitable for complex application scenarios in real environments, thereby improving the test effect.

[0031] In addition, the dependency graph model constructed in the present invention can better reflect the dependency between operations, improve the quality of generated test cases from both the operation and parameter aspects, and based on a good feedback mechanism, can reduce the number of erroneous parameters, further reduce the number of generated test cases, and avoid the problem of excessive test case size; at the same time, generate test subgraphs as test cases to reflect more complex logical relationships between operations, which can improve the effectiveness of test cases and thereby improve the coverage of test targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of a test case generation method based on a dependency graph provided by the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0034] In a first aspect, the present invention provides a test case generation method based on a dependency graph.

[0035] The present invention provides a test case generation method based on dependency graph, which uses a dependency graph model to generate test cases. Different from the common graph model structure, the present invention adopts a special directed graph structure as the basic structure of the dependency graph model. The model stores the production-consumption relationship between operations and parameters, and also stores the relevant data of operations and parameters. After obtaining the dependency graph (operation-parameter graph, OPG), a stateful test case generation algorithm is used to generate the final test case set ( Figure 1 In the example, get and post represent the types of operations, and a simplified test subgraph is composed of two get operations and one post operation).

[0036] See also Figure 1 As shown, the present invention provides a test case generation method based on a dependency graph, and its specific implementation process is as follows:

[0037] Step S1: construct an adaptive dependency graph model;

[0038] Some existing graph models use parameters as edges and operations as nodes to construct graph models. This type of graph model has the problem that the same parameter is represented by multiple edges, so parameter information cannot be synchronized in time and is difficult to manage. In some studies, graph models are constructed with schema and operations as nodes, where schema nodes represent resources, which will lead to problems such as too few nodes and unclear dependencies, and further lead to problems such as few generated test sequences and incomplete tests. Therefore, based on the above problems, the present invention designs an adaptive dependency graph model with parameters and operations as nodes. First, using parameters as nodes can solve the problem that parameter information cannot be synchronized in existing graph models, and at the same time can reflect the logical relationship between parameters and operations at a fine-grained level.

[0039] The dependencies between API operations in cloud services are complex, and the number of endpoints ranges from dozens to hundreds. The present invention proposes to use a graph structure to store the logical relationship between parameters and operations, while solving the problem of lack of global vision when generating stateful test cases.

[0040] Based on the characteristics of RESTful API, the present invention proposes a dependency graph model OPG = (V, E, μ), in which the following relationship exists:

[0041] (1) V = V parameter ∪V operation , V represents a node set, which includes an operation node set V operation and parameter node set V parameter Two categories;

[0042] (2) E=E op ∪E po , E represents a set of directed edges, including the set of edges E that operate to the parameters op The set of edges E pointing to the operation with parameters po Two categories;

[0043] Specifically, an edge from an operation to a parameter indicates that the operation can produce the parameter, and an edge from a parameter to an operation indicates that the operation can consume the parameter;

[0044] (3) μ = μ parameter ∪μ operation , μ represents a set of mapping functions between operations and parameters adding attribute values, where μ parameter Indicates the function of adding attributes to parameters, μ operation A function representing an operation that adds attributes.

[0045] Step S2: Establish a stateful test case generation algorithm;

[0046] After constructing the dependency graph model, a stateful test case is generated according to the dependency relationship between the parameters and operations shown in the dependency graph model, which includes two steps: the generation of the test subgraph and the generation of test parameters. The first is the generation of the test subgraph. By traversing each operation node in the dependency graph model, a test subgraph is generated for the operation node according to the reverse generation algorithm. The test subgraph includes the operations involved in a single test in the test case, as well as the order in which each operation is executed; then the test parameters are generated. The test parameters are generated in real time during the test process by combining multiple generation strategies. The success value, response value, sample value / default value, and random generation strategies are used to generate specific test parameter values ​​for the parameters involved in the test case according to the descending order of the generation strategy priority. The test subgraph and the test parameters together constitute the final test case. After traversing the entire dependency graph, a set of test cases in a test is obtained.

[0047] Specifically, the generation of the test subgraph is implemented by using the OPG traversal algorithm and the reverse generation algorithm. Specifically, the test subgraph can be generated by traversing the dependency graph according to the reverse generation algorithm. The specific implementation process is as follows:

[0048] First, initialize a test subgraph and sequence, set the node a to be tested as the root node of the test subgraph, and add it to the sequence. In the dependency graph model, access the parent node of node a to obtain the parameter value required for the operation, and access the parent node of each parameter value to obtain the operation set that can produce the parameter. Among these operations, select the operations that satisfy the producer-consumer relationship, CRUD semantic constraints, and do not form a loop, and add the parameter nodes and operation nodes to the test subgraph in turn. The operation nodes are also appended to the front of the sequence in turn, and the above operations are repeated until all the operation nodes in the sequence are visited.

[0049] Specifically, test parameters are generated in real time during the test process by combining multiple generation strategies to form a test case set. The specific implementation process is as follows:

[0050] The generation strategies used are sorted in order of optimization level: success value generation strategy, response value generation strategy, sample value / default value generation strategy, and random value generation strategy. First, the success value generation strategy is selected. If there is no success value, it is obtained from the response value. These generation strategies are applied in descending order according to the optimization level until the value range of each test parameter is determined.

[0051] The following is a detailed description of each generation strategy:

[0052] (1) Success value: the value saved in the previous successful test case;

[0053] (2) Response value: the value obtained by analyzing the test feedback message; it is divided into real-time response value and total response value, of which the real-time response value is optimized;

[0054] (3) Example value / default value: The value obtained from the analysis of the RestfulAPI specification document. The example value and the default value generally do not appear at the same time, so they have the same priority. If they appear at the same time, just randomly select one;

[0055] (4) Random value: When the above success value, response value, sample value / default value cannot obtain valid data values, data is randomly generated according to the parameter type, format and other requirements.

[0056] The present invention traverses the operation nodes based on the constructed dependency graph, and generates a test subgraph as a test case for each operation node according to its dependency. There are two advantages of using the test subgraph as a test case. First, the test subgraph retains the production and consumption paths between operations and parameters, and the test results can be analyzed in real time during the test process, and the analysis results are updated in real time to the dependency graph to optimize the test effect. Second, the logical relationship reflected by the test subgraph is more complex, not limited to linear logical relationships, and can reflect the logical relationship between complex operations in the actual test environment, thereby improving the test effect.

[0057] In a second aspect, the present invention provides a test case generation system based on a dependency graph, for implementing a test case generation method based on a dependency graph provided in the first aspect.

[0058] The invention discloses a test case generation system based on a dependency graph, mainly comprising: an adaptive dependency graph model construction module, a test subgraph generation module and a test parameter generation module; wherein the adaptive dependency graph model construction module is mainly used to construct an adaptive dependency graph model, in which the production-consumption relationship between operations and parameters is stored, and the relevant data of the operations and parameters are stored; the test subgraph generation module mainly comprises an OPG traversal algorithm module and a reverse generation algorithm module, the OPG traversal algorithm module is mainly used to traverse each operation node in the dependency graph model, and the reverse generation algorithm module is mainly used to generate a test subgraph according to the operation node, and the test subgraph includes the operations involved in a single test in the test case, and the execution order of each operation; the test parameter generation module is mainly used to adopt a method combining multiple generation strategies to generate test parameters in real time during the test process, and the generation strategies adopted are arranged in order according to the optimization level: success value generation strategy, response value generation strategy, example value / default value generation strategy, and random value generation strategy. First, the success value generation strategy is selected, and if there is no success value, it is obtained from the response value, and these generation strategies are applied in descending order according to the optimization level until the value range of each test parameter is determined.

[0059] In a third aspect, the present invention provides an electronic device for implementing the steps of a test case generation method based on a dependency graph provided in the first aspect.

[0060] An electronic device of the present invention specifically includes the following components: a processor, a memory, a communication interface and a communication bus.

[0061] The processor, memory and communication interface provided in the present invention communicate with each other via a communication bus.

[0062] The specific functions and effects of the processor, memory, communication interface and communication bus provided in the present invention are:

[0063] The processor is mainly used to execute the computer program stored in the memory to implement the steps of a test case generation method based on a dependency graph provided in the first aspect.

[0064] Preferably, the processor can specifically adopt a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also adopt a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0065] Memory is mainly used to store computer programs and local data uploaded by users.

[0066] Preferably, the memory may be a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. At the same time, the memory may also be at least one storage device located away from the processor.

[0067] The communication interface is mainly used for communication between an electronic device of the present invention and other devices.

[0068] The communication bus is mainly used to realize communication between the processor, communication interface and memory.

[0069] Preferably, the communication bus may specifically adopt a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc., and the communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0070] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A test case generation method based on a dependency graph, characterized in that: The following steps are involved: Step S1: construct an adaptive dependency graph model; The dependency graph model is: OPG = (V, E, μ); V represents a node set, V = V parameter ∪V operation , V operation Represents the set of operation nodes, V parameter represents the parameter node set; E represents the directed edge set, E=E op ∪E po , E op represents the set of edges from the operation to the parameter, E po represents the edge set from which parameters point to operations; μ represents the set of mapping functions between operations and parameter attribute values, μ = μ parameter ∪μ operation , μ parameter Represents a function that adds attributes to a parameter, μ operation A function that represents the operation of adding attributes; Step S2: Establish a stateful test case generation algorithm; S2.1: Generate test subgraphs using OPG traversal algorithm and reverse generation algorithm; S2.2: A combination of multiple generation strategies is used to generate test parameters in real time during the test process.

2. A test case generation method based on dependency graph according to claim 1, characterized in that: In step S2.1, a test subgraph and sequence are initialized, the node a to be tested is set as the root node of the test subgraph, and added to the sequence.

3. A test case generation method based on dependency graph according to claim 2, characterized in that: In step S2.1, the parent node of node a is accessed in the dependency graph model to obtain the parameter value required for the operation, and the parent node of each parameter value is accessed to obtain the operation set that can produce the parameter.

4. A test case generation method based on dependency graph according to claim 3, characterized in that: In step S2.1, operations that satisfy the producer-consumer relationship, CRUD semantic constraints and do not form a loop are selected, and parameter nodes and operation nodes are added to the test subgraph in turn. The operation nodes are also appended to the front of the sequence in turn, and the above operations are repeated until all operation nodes in the sequence are visited.

5. The test case generation method based on dependency graph according to claim 1, characterized in that: In step S2.2, the generation strategies include: a success value generation strategy, a response value generation strategy, an example value / default value generation strategy and a random value generation strategy.

6. A test case generation method based on dependency graph according to claim 5, characterized in that: The generation strategies are arranged in order according to the optimization level: success value generation strategy, response value generation strategy, example value / default value generation strategy, and random value generation strategy.

7. The test case generation method based on dependency graph according to claim 5, characterized in that: In step S2.2, a success value generation strategy is first selected. If no success value exists, it is obtained from the response value. These generation strategies are applied in descending order according to the optimization level until the value range of each test parameter is determined.

8. The test case generation method based on dependency graph according to claim 5, characterized in that: The success value is the value saved in the previous successful test case; the response value is the value analyzed from the test feedback message, which is divided into real-time response value and total response value; the example value / default value is the value analyzed from the RestfulAPI specification document; the random value is: when the above success value, response value, example value / default value cannot obtain valid data values, the data is randomly generated according to the parameter requirements.

9. A test case generation system based on dependency graph, characterized in that: A test case generation method based on a dependency graph is used to implement any one of claims 1-8, the system comprising: an adaptive dependency graph model construction module, a test subgraph generation module and a test parameter generation module; the adaptive dependency graph model construction module is used to construct an adaptive dependency graph model; the test subgraph generation module comprises an OPG traversal algorithm module and a reverse generation algorithm module, the OPG traversal algorithm module is used to traverse each operation node in the dependency graph model, and the reverse generation algorithm module is used to generate a test subgraph according to the operation node; the test parameter generation module is used to generate test parameters in real time during the test process by combining multiple generation strategies.

10. An electronic device, characterized in that: include: Memory, used to store computer programs and local data uploaded by users; A processor, configured to execute a computer program stored in a memory to implement the steps of a test case generation method based on a dependency graph as described in any one of claims 1 to 8; Communication interface, used for communication between electronic equipment and other devices; The communication bus is used to realize communication between the processor, the communication interface and the memory.

Images