Parallel test system based on large model

Abstract

The invention relates to a parallel test system based on a large model, which is characterized by comprising a test case analysis system and a case dependency relationship model service, the case dependency relationship model service comprises an inter-case dependency relationship service, and the inter-case dependency relationship service comprises an inter-case dependency relationship service interface and a prediction model. According to the method, the dependency relationship among the use cases is accurately obtained through a large model technology, test use cases or processes without the dependency relationship can be better scheduled and executed in parallel based on the dependency relationship, test resources are utilized to the maximum extent, and repeated operation can be avoided when a plurality of use cases have the same preconditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of parallel testing systems, and in particular relates to a parallel testing system based on a large model. Background Art

[0002] In order to ensure the quality of the system, a large number of automated use cases need to be maintained. A large number of regression use cases generally take a long time to execute. To improve the execution efficiency, a subset of independent use cases can be executed in parallel under the premise of dependencies between use cases. The traditional method is to manually configure the dependencies between use cases, but this method requires manual maintenance, and when there are many team members, it is usually difficult for individuals to manually maintain dependencies to cope with the correct and comprehensive dependencies between use cases caused by the rapidly expanding test cases of the team, and it is also impossible to effectively reduce the execution time of the full set of use cases. Therefore, a means of accurately obtaining the dependencies between use cases is needed, based on which test tasks can be better scheduled and executed in parallel, thereby reducing the cost of executing a large number of test cases. Summary of the invention

[0003] (I) Purpose of the invention

[0004] In order to overcome the above shortcomings, the purpose of the present invention is to provide a parallel testing system based on a large model to solve the above technical problems.

[0005] (II) Technical solution

[0006] To achieve the above objectives, the technical solutions provided by this application are as follows:

[0007] A parallel testing system based on a large model includes a test case analysis system and a test case dependency model service, wherein the test case dependency model service includes an inter-case dependency service interface and a prediction model.

[0008] Dependency graph creation: The test case analysis system calls the dependency service interface between test cases to obtain the dependencies between all test cases and create multiple independent dependency graphs, and generate a directed acyclic graph. The test cases in each graph have dependencies, and there are no dependencies between graphs.

[0009] Topological sorting to obtain independent test sets: Based on the directed acyclic graph of the dependency relationships between use cases, use the topological sorting results to obtain independent use case sets that can be executed in sequence;

[0010] For manual test case sets, a user-friendly front-end system is provided based on the interface return, which can be used as a reference for testers when dividing the work.

[0011] For automated test suites, receive independent test suite data from the test case analysis system and use this data in conjunction with Kubernetes to dynamically create and manage the test suite execution environment.

[0012] Monitor the efficiency of parallel test execution and collect data on test time and resource usage.

[0013] Preferably, the prediction model includes:

[0014] Data collection: manually analyze and annotate the dependencies between test case pairs. Dependency data can also be automatically generated through some empirical rules.

[0015] Data formatting, input is a use case pair, key is the use case id, value is the text description or test code of the two use cases, and output is a Boolean value, indicating whether there is a dependency relationship;

[0016] Use the big model to directly analyze and identify the dependencies between test cases. By providing the big model with test case pair data and asking it to determine whether there is a dependency, preliminary results can be obtained.

[0017] Fine-tune the large model, using GPT-4 as the basis of the pre-trained model, setting the fine-tuning parameters of learning rate, training rounds, and batch size, and fine-tuning the large model using the prepared custom dataset;

[0018] Optimize and monitor, continuously monitor the prediction accuracy of the model, collect data with prediction errors, regularly use new labeled data to fine-tune the model to improve the model's prediction performance, feed the prediction error data back to the labeling process, and improve the data set.

[0019] Preferably, the fine-tuned model is deployed as an API service that accepts input test case pairs and returns dependency prediction results.

[0020] Preferably, the dependency graph creation includes receiving all test cases, for each unvisited test case, calling the test case dependency model service to obtain the dependency between each pair of test cases, and using breadth-first search traversal to construct its dependency graph,

[0021] Preferably, the dependencies are stored in a dictionary format, where each key is a test case and the corresponding value is a list of test cases that depend on it.

[0022] Preferably, the operation steps of topological sorting to obtain independent test sets are specifically as follows: first, the in-degree of each test case is calculated, and a queue is used to store test cases with an in-degree of zero. The in-degree represents the number of test cases that depend on the test case, and the in-degree of its adjacent nodes is reduced. If the in-degree of an adjacent node is reduced to zero, it is added to the queue, and the above steps are repeated until all nodes are processed. In addition, interfaces can be provided to return multiple independent test case sets that are executed in sequence according to the test case type.

[0023] Preferably, use the Kubernetes API to create a client instance to perform kubernetes operations, define a function, receive a set of use cases to dynamically create a corresponding deploymentmnet in Kubernetes, and ensure that each independent test set has a unique deploymentmnet name. According to the Kubernetes API requirements, it is necessary to build a dictionary object containing the Deployment configuration information. The dictionary defines the relevant attributes of the deployment, which are name, number of replicas, and container specifications. Finally, a request is made to Kubernetes for deployment. Each container contains an independent set of use cases, an operating system execution environment, and a system under test execution environment.

[0024] Beneficial effects:

[0025] The present invention uses large model technology to accurately obtain the dependencies between test cases, based on which test cases or processes without dependencies can be better scheduled and executed in parallel, maximizing the use of test resources, and avoiding repeated operations when multiple test cases have the same preconditions. The large model is used to obtain the dependencies between test cases, and a system that can execute test cases in parallel is designed based on this, which significantly reduces the execution time of test tasks, and can reduce various costs while ensuring software quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the process of the present invention;

[0027] Figure 2 It is a use case dependency model service diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following is a detailed description of the present invention in conjunction with the specific implementation methods and with reference to the attached drawings. Figure 1-2 , the present invention is further described in detail. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0029] It includes a test case analysis system 100 and a use case dependency model service 200. The use case dependency model service 200 includes an inter-use case dependency service including an inter-use case dependency service interface and a prediction model.

[0030] The test case analysis system 100 calls the dependency prediction model service between test cases, obtains the dependencies between all test cases and creates multiple independent dependency graphs, that is, the test cases in each graph are dependent, and there is no dependency between graphs. For an independent directed acyclic graph, if there is an edge from node A to node B, it means that use case B depends on use case A, that is, B must be executed after A is executed. Specifically, all test cases are received, and for each test case that has not been visited, the dependency prediction model service between test cases is called to obtain the dependencies between each pair of test cases, and the dependency graph is constructed using a breadth-first search traversal. Dependencies are stored in the form of a dictionary, where each key is a test case and the corresponding value is a list of test cases that depend on it.

[0031] Based on the directed acyclic graph of the dependency relationship between use cases, use the topological sorting results to obtain independent sets of use cases that can be executed in sequence. Specifically, first calculate the in-degree of each test case, which indicates the number of test cases that depend on it, and use a queue to store test cases with an in-degree of zero, that is, test cases with no dependencies, and reduce the in-degree of the adjacent nodes of the test cases that depend on it. If the in-degree of the adjacent node is reduced to zero, add it to the queue and repeat the above steps until all nodes are processed. Topological sorting ensures the correct execution order between use cases. In addition, interfaces can be provided to return multiple independent test case sets that are executed in sequence according to their use case types.

[0032] For manual test case sets, a display-friendly front-end system is provided based on the interface return, which can be used as a reference for testers when dividing work.

[0033] For automated test case sets, receive independent test case set data from the test case analysis system 100, and use this data in combination with Kubernetes to dynamically create and manage the execution environment of the test case set. Specifically, use the Kubernetes API to create a client instance to perform kubernetes operations, such as creating containers for deployment, obtaining resource status, etc. Define a function to receive a set of test cases to dynamically create a corresponding deploymentmnet in Kubernetes, ensuring that each independent test set has a unique deploymentmnet name. According to the API requirements of Kubernetes, it is necessary to build a dictionary object containing Deployment configuration information. The dictionary defines the relevant attributes of the deployment, such as name, number of replicas, container specifications, etc. Finally, a request is made to Kubernetes for deployment. Each container contains an independent set of test cases, an operating system, and a related execution environment such as the system under test.

[0034] Monitor the efficiency of parallel test execution and collect data on test time and resource usage.

[0035] The use case dependency model service 200 includes:

[0036] 1. Data collection: Manually analyze and annotate the dependencies between test case pairs, or automatically generate dependency data through some empirical rules.

[0037] 2. Data formatting: the input is a use case pair, the key is the use case id, the value is the text description or test code of the two use cases, and the output is a Boolean value, indicating whether there is a dependency relationship.

[0038] 2. Use large models to directly analyze and identify dependencies between test cases. Preliminary results can be obtained by providing test case pair data to large models such as GPT-4 and asking them to determine whether there is a dependency relationship.

[0039] 3. Fine-tune the large model. For example, GPT-4 is used as the basis of the pre-trained model, and fine-tuning parameters such as learning rate, training rounds, batch size, etc. are set, and the large model is fine-tuned using the prepared custom dataset.

[0040] 4. Deploy the fine-tuned model as an API service that accepts input test case pairs and returns dependency prediction results.

[0041] 5. Optimization and monitoring. Continuously monitor the prediction accuracy of the model and collect data with prediction errors. Regularly use new annotated data to fine-tune the model and improve the model's prediction performance. Feed the prediction error data back into the annotation process to improve the data set.

[0042] The present invention takes the text description of manual test cases and the annotations and codes of automated test cases as input, and directly extracts the dependencies between test cases by using the natural language understanding and code understanding capabilities of the large model;

[0043] In order to make the model better understand the dependencies and context between test cases and improve the accuracy of the model, fine-tune it using the existing test case dependency data. Collect training data: annotate a batch of known test case dependencies; fine-tune the large model: initialize the model, customize the data set, set the training parameters, and train the model. Use the fine-tuned model to more accurately determine the dependencies between test case pairs.

[0044] Use a unified format that includes test cases and their dependencies to facilitate subsequent use and management.

[0045] Multiple independent directed acyclic graphs are generated from the dependencies between use cases, and the directed acyclic graphs of each use case set are topologically sorted, ensuring the correctness of the use case dependency order of a single test set. Each independent use case set can be executed in parallel.

[0046] The present invention implements a service that can dynamically receive test case set requests from a test case analysis system, and is used to create and manage the execution environment of these test cases in Kubernetes, which not only makes the deployment of test cases automated and scalable, but also improves the efficiency and flexibility of the overall test.

[0047] The present invention uses large model technology to accurately obtain the dependencies between test cases, based on which test cases or processes without dependencies can be better scheduled and executed in parallel, maximizing the use of test resources, and avoiding repeated operations when multiple test cases have the same preconditions. The large model is used to obtain the dependencies between test cases, and a system that can execute test cases in parallel is designed based on this, which significantly reduces the execution time of test tasks, and can reduce various costs while ensuring software quality.

[0048] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A parallel testing system based on a large model, characterized in that: It includes a test case analysis system and a use case dependency model service. The use case dependency model service includes an inter-use case dependency service interface and a prediction model. Dependency graph creation: The test case analysis system calls the dependency service interface between test cases to obtain the dependencies between all test cases and create multiple independent dependency graphs, and generate a directed acyclic graph. The test cases in each graph have dependencies, and there are no dependencies between graphs. Topological sorting to obtain independent test sets: Based on the directed acyclic graph of the dependency relationships between use cases, use the topological sorting results to obtain independent use case sets that can be executed in sequence; For manual test case sets, a user-friendly front-end system is provided based on the interface return, which can be used as a reference for testers when dividing the work. For automated test suites, receive independent test suite data from the test case analysis system and use this data in conjunction with Kubernetes to dynamically create and manage the test suite execution environment. Monitor the efficiency of parallel test execution and collect data on test time and resource usage.

2. A large model-based parallel testing system according to claim 1, characterized in that: Predictive models include: Data collection: manually analyze and annotate the dependencies between test case pairs. Dependency data can also be automatically generated through some empirical rules. Data formatting, input is a use case pair, key is the use case id, value is the text description or test code of the two use cases, and output is a Boolean value, indicating whether there is a dependency relationship; Use the big model to directly analyze and identify the dependencies between test cases. By providing the big model with test case pair data and asking it to determine whether there is a dependency, preliminary results can be obtained. Fine-tune the large model, using GPT-4 as the basis of the pre-trained model, setting the fine-tuning parameters of learning rate, training rounds, and batch size, and fine-tuning the large model using the prepared custom dataset; Optimize and monitor, continuously monitor the prediction accuracy of the model, collect data with prediction errors, regularly use new labeled data to fine-tune the model to improve the model's prediction performance, feed the prediction error data back to the labeling process, and improve the data set.

3. A large model-based parallel testing system according to claim 2, characterized in that: The fine-tuned model is deployed as an API service that accepts input test case pairs and returns dependency prediction results.

4. The large model-based parallel testing system according to claim 1, characterized in that: Dependency graph creation includes receiving all test cases, and for each unvisited test case, calling the use case dependency model service to obtain the dependency between each pair of test cases, and using breadth-first search traversal to build its dependency graph.

5. A large model-based parallel testing system according to claim 4, characterized in that: Dependencies are stored in the form of a dictionary, where each key is a test case and the corresponding value is a list of test cases that depend on it.

6. A large model-based parallel testing system according to claim 1, characterized in that: The specific operation steps of topological sorting to obtain independent test sets are as follows: first calculate the in-degree of each test case, and use a queue to store test cases with an in-degree of zero. The in-degree represents the number of test cases that depend on the test case, and reduce the in-degree of its adjacent nodes. If the in-degree of an adjacent node is reduced to zero, add it to the queue and repeat the above steps until all nodes are processed. In addition, interfaces can be provided to return multiple independent test case sets that are executed in sequence according to the test case type.

7. The large model-based parallel testing system according to claim 1, characterized in that: Use the Kubernetes API to create a client instance to perform kubernetes operations, define a function, receive a set of use cases to dynamically create a corresponding deploymentmnet in Kubernetes, and ensure that each independent test set has a unique deploymentmnet name. According to the Kubernetes API requirements, it is necessary to build a dictionary object containing the Deployment configuration information. The dictionary defines the relevant attributes of the deployment, which are name, number of replicas, and container specifications. Finally, a request is made to Kubernetes for deployment. Each container contains an independent set of use cases, an operating system execution environment, and a system under test execution environment.

Images