Test case generation method and device, equipment, program product and storage medium

Abstract

The invention provides a test case generation method and device, equipment, a program product and a storage medium. The method comprises the steps of generating a plurality of first test cases based on a first mode, and adding the first test cases to a test case database; under the condition that the current state of the test case database meets the first condition, generating a plurality of second test cases based on a second mode; storing the plurality of second test cases in a test case database; under the condition that the current state of the test case database meets a second condition, generating a test case of the test case database based on a third mode; and under the condition that the current state of the test case database meets a third condition, executing the step of generating a plurality of first test cases based on the first mode.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a test case generation method, device, equipment, program product and storage medium. Background Art

[0002] As the scale of software development increases and business requirements become more complex, the defects of use cases in the software testing process are gradually becoming more prominent. In related technologies, the coverage of traditional manually written use cases is low and highly dependent on the experience of testers. In the early stages of software projects, the use of other businesses, document organization and manual use case generation increase labor costs, maintenance costs and time costs. The generated test cases have low output, low coverage and low accuracy, resulting in repeated defects. Summary of the invention

[0003] The embodiments of the present application provide a test case generation method, apparatus, device, program product and storage medium, which can improve the accuracy of test case generation.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for generating a test case, the method comprising:

[0006] Generate multiple first test cases based on the first method;

[0007] adding the plurality of first test cases to an initial test case database;

[0008] When the current state of the test case database satisfies the first condition, generating a plurality of second test cases based on a second method;

[0009] Storing the plurality of second test cases in the test case database;

[0010] When the current state of the test case database satisfies the second condition, generating test cases for the test case database based on a third method;

[0011] When the current state of the test case database satisfies the third condition, the step of generating a plurality of first test cases based on the first method is performed.

[0012] The present application provides a test case generation device, comprising:

[0013] A first test case generation module, configured to generate a plurality of first test cases based on a first method; and add the plurality of first test cases to a test case database;

[0014] A second test case generation module is used to generate a plurality of second test cases based on a second method when the current state of the test case database satisfies the first condition; and store the plurality of second test cases in the test case database;

[0015] A third use case generation module is used to generate test cases for the test case database based on a third method when the current state of the test case database meets the second condition; and to execute the step of generating multiple first test cases based on the first method when the current state of the test case database meets the third condition.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the test case generation method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the test case generation method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the test case generation method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By generating multiple first test cases based on the first method and adding them to the test case database, determining the currently applicable test case generation method according to the current state of the test case database, and switching the generation method according to different states of the test case database, the test cases are matched with the current requirements of the test case library, which can improve the quality and accuracy of generated test cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an application mode of the test case generation method provided in an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0025] Figure 3A It is a first flow chart of the test case generation method provided in an embodiment of the present application;

[0026] Figure 3B It is a second flow chart of the test case generation method provided in an embodiment of the present application;

[0027] Figure 3C It is a third flow chart of the test case generation method provided in the embodiment of the present application;

[0028] Figure 3D It is a fourth flow chart of the test case generation method provided in an embodiment of the present application;

[0029] Figure 3E It is a fifth flow chart of the test case generation method provided in the embodiment of the present application;

[0030] Figure 3F It is a sixth flow chart of the test case generation method provided in an embodiment of the present application;

[0031] Figure 3G This is a seventh flow chart of the test case generation method provided in an embodiment of the present application;

[0032] Figure 4 This is a schematic diagram of the eighth flow chart of the test case generation method provided in the embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of the overall process of the project use case generation method provided in the embodiment of the present application;

[0034] Figure 6 It is a flowchart of a method for generating use cases in the early stage of a project provided by an embodiment of the present application;

[0035] Figure 7 It is a flowchart of a method for generating a use case in the mid-term stage of a project provided in an embodiment of the present application;

[0036] Figure 8 It is a flowchart of a method for generating use cases in the later stage of a project provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) Project: is the entire software development project or product development project, including the planning and management of the overall project goals, scope, requirements, schedule, resource allocation, etc. In the embodiments of the present application, the projects executed at different stages represent software development projects.

[0044] 2) Continuous Integration and Continuous Delivery (CICD): A method of frequently delivering applications to customers by introducing automation in the application development phase. Continuous integration is a process in which new changes to the code are regularly built, tested, and merged into the main code base to quickly detect and resolve integration issues. Continuous delivery is an automated build, test, and deployment process that enables software to be delivered to users at any time.

[0045] 3) Large Language Model (LLM): It is an artificial intelligence algorithm used to predict human language, including text, spoken language and gestures.

[0046] 4) Business changes: In the embodiments of the present application, it refers to changes to the original functional requirements and codes.

[0047] 5) Business keywords: refers to the keywords used in the business field to describe and identify business concepts, business entities, business processes, business activities, etc.

[0048] 6) Defect Escape Rate (DER): The ratio of the number of defects still existing in the software product after a specific testing phase to the number of all known defects before the same phase, indicating the proportion of defects that were not discovered during the testing phase.

[0049] 7) Bidirectional and Auto-Regressive Transformers (BART): This is a transformer model that combines bidirectional and autoregressive properties, hereinafter referred to as the BART model, which generates a labeled output sequence each time, conditioned on the previously generated labels and the input provided by the encoder. Through pre-training and multi-task learning on a large-scale corpus, the BART model has powerful language understanding and generation capabilities and can be used for natural language processing tasks such as text generation, summarization, and question answering.

[0050] 8) Adaptive moment estimation algorithm (Adam): A commonly used optimization algorithm used for training neural network models, hereinafter referred to as the Adam algorithm. The Adam algorithm combines the concepts of gradient descent and momentum, and is adaptive. It can update model parameters more effectively, speed up training, and has different learning rates for different parameters.

[0051] 9) F1 value: A performance metric used to evaluate the balance between accuracy and recall (precision) of a classification model.

[0052] 10) Data noise: It is the errors, interference, inconsistency or inaccurate information present in the text dataset.

[0053] In the early stage of test case generation in related technologies, reference is usually made to other businesses. Complicated businesses rely on manual case writing with low coverage, are highly dependent on the experience of testers, have high labor costs, maintenance costs and time costs, and have a low output ratio. The accuracy and coverage of generated case studies are not high.

[0054] Embodiments of the present application provide a test case generation method, a test case generation device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of generating test cases.

[0055] The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The device provided in the embodiments of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal or a server.

[0056] See also Figure 1 , Figure 1 It is a schematic diagram of the application mode of the test case generation method provided in the embodiment of the present application, which is an example of supporting a test case generation application. Figure 1 The system involves a server 200, a network 300, a terminal device 400 and a database 500. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] In some embodiments, the user is a tester, the server 200 is a server that executes a test case generation method, the terminal device 400 can be a computer or a mobile phone of the tester, the terminal device 400 is used to monitor business changes and notify business changes, and the database 500 stores initial test cases and test cases generated in different project stages.

[0058] For example, terminal device 400 is used to monitor business changes. When a business change occurs, the tester is notified. The tester creates corresponding artificial use cases based on the changed business. Terminal device 400 transmits the business changes and artificial use cases to server 200 through network 300. Server 200 executes the use case generation method of different stages of the project, generates test cases and stores them in database 500. Server 200 feeds back the test results of the application to be tested to terminal device 400 through network 300.

[0059] In some embodiments, the test case generation method of the embodiment of the present application can also be applied in the following application scenarios: testing of applications under development, for example: developing a new application, in order to test whether there are defects in the newly developed application, it is necessary to design test cases for the application for testing. Since the application is software in a new field, there are few use cases that can be referenced in the early stage of test case generation. The software use case generation method of the embodiment of the present application can be supplemented by generating use cases based on a corpus related to the reference use cases, iteratively generating high-quality test cases based on the test case recommendation model and the large language model, improving the coverage of the generated use cases, and assisting in testing the defects of the application under development.

[0060] The embodiments of the present application can be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0061] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0062] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0063] Large Language Model (LLM): A machine learning model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data. They are usually based on deep learning architectures. Large language models refer to deep learning models trained with large amounts of text data, containing billions or even larger parameters. They can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities.

[0064] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the server provided in the embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 410, a memory 450 and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0066] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0068] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0072] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The test case generating device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a first test case generating module 4551, a second test case generating module 4552 and a third test case generating module 4553. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the test case generating device 455 excludes the implementation that can only include the first case generating module 4551. The functions of each module will be explained below.

[0073] The test case generation method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server / terminal device provided in the embodiment of the present application.

[0074] The following describes the test case generation method provided by the embodiment of the present application. As mentioned above, the electronic device implementing the test case generation method of the embodiment of the present application may be a server. Therefore, the execution subject of each step will not be repeatedly described below.

[0075] See also Figure 3A , Figure 3A This is a first flow chart of the test case generation method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0076] In step 301, a plurality of first test cases are generated based on a first method.

[0077] For example, a project is an entire software development or product development project. In the embodiments of the present application, projects executed at different stages represent software development projects. A software project may have defects during the development process. In order to correct the defects, it is necessary to design relevant test cases for the project's requirement documents and code data to test the project and discover defects in the project development process. The first method is a method for generating test cases in the early stages of a project. In the early stages of a project, the number of basic requirement documents, codes, and use cases is insufficient. At this time, multiple first test cases generated based on the first method are used to supplement the use cases in the early stages of the project.

[0078] For example, the test case generation method provided in the embodiment of the present application is described by taking the application scenario of the development process of an application as an example. The continuous integration / continuous delivery process will perform real-time detection on each stage of the project in the embodiment of the present application. When the business change function in the continuous integration / continuous delivery process monitors changes in the project's requirements, codes, and documents, the continuous integration / continuous delivery process is automatically triggered, and the changes are notified to the relevant testers through the message notification module, so as to facilitate the acquisition of relevant artificial use cases. Continuous integration / continuous delivery is a business model that monitors business changes in the development process in real time, and reminds relevant personnel through message notifications when changes occur. In the embodiment of the present application, different stages of the project represent different stages of the software development project.

[0079] In some embodiments, see Figure 3B , Figure 3B It is a second flow chart of the test case generation method provided in an embodiment of the present application; step 301 can be implemented through steps 3011 to 3012, which are described in detail below.

[0080] In step 3011, data cleaning is performed on the first corpus database to obtain a basic database.

[0081] Here, the first mode is used to indicate generating a test case based on the first corpus.

[0082] For example, the first corpus database includes the following types of data: candidate test cases, requirement texts, and code data of the application to be tested. In the early stages of a project, the initial test case database contains less content and cannot meet the requirements for automatically writing and generating test cases. At this time, a first corpus database related to test cases is obtained, and the requirement texts, code data, and use cases related to test cases in the first corpus database are referenced for the generation of test cases in the early stages of the project. The first corpus database can be a development case library related to the entire network and application (for example, in actual applications, it can be a case library of a company that develops an application), a requirement library, and a code library as data sources.

[0083] For example, data cleaning is performed on the data in the first corpus database. Data cleaning is the operation of deduplication, standardization, and data set segmentation and analysis. Deduplication is to delete duplicate data items in the first corpus database; standardization is to unify the data in the first corpus database to eliminate the differences between different data types and units; data set segmentation is the process of dividing the first corpus database into a training set, a validation set, and a test set.

[0084] See also Figure 6 , Figure 6 yes Figure 6 It is a flowchart of a method for generating use cases in the early stage of a project provided by an embodiment of the present application; Figure 6 The use case recommendation model 620 includes a data processing model 621, which calls the data processing model 621 to perform data cleaning on the first corpus database to complete the data cleaning process, that is, to complete the preprocessing of the first corpus database to obtain a basic database.

[0085] In an embodiment of the present application, data cleaning processing is performed on the first corpus database to eliminate duplicate information and avoid misleading or unnecessary duplication of work in the subsequent test case generation process. Standardization processing can ensure data consistency and make the data more accurate and reliable. The division of the first corpus database provides effective training data and evaluation model basis for subsequent models, ensuring the accuracy and stability of the generated test cases.

[0086] Continue to see Figure 3B In step 3012, a plurality of first test cases are generated based on the basic database.

[0087] For example, after obtaining the basic database after data cleaning, the use case recommendation model is called to recommend test cases, and multiple first test cases are obtained, that is, multiple final use cases in the early stages of the project are obtained. To perform test case recommendation processing on the basic database, it is necessary to first matrix the data in the basic database, extract eigenvalues ​​and eigenvectors, determine the corresponding similarity matrix, perform matrix decomposition on the similarity matrix, and determine the final use case, that is, the first test case, based on the similarity result of the matrix decomposition.

[0088] In some embodiments, see Figure 3C , Figure 3C This is the third flow chart of the test case generation method provided in the embodiment of the present application; step 3012 can be implemented through steps 30121 to 30123, which are described in detail below.

[0089] In step 30121, feature extraction processing is performed on each type of data in the basic database to obtain the first candidate test case feature of each candidate test case, the first requirement feature of each requirement text, and the first code feature of each code data.

[0090] For example, the data in the basic database is matrixed and the eigenvalues ​​and eigenvectors are extracted. The test case recommendation model is called based on the basic database to perform case recommendation. Figure 6 , Figure 6 It is a flow chart of the use case generation method in the early stage of the project provided by the embodiment of the present application; the called use case recommendation model 620 is composed of a data processing model 621, a matrix decomposition recommendation model 622 and a collaborative recommendation filtering model 623. After the data processing model 621 performs data cleaning on the first corpus database to obtain a basic database, the matrix decomposition recommendation model 622 is called to perform feature extraction processing on each type of data in the basic database to obtain the first candidate test case feature of each candidate test case, the first requirement feature of each requirement text, and the first code feature of each code data.

[0091] In step 30122, based on the similarity between each first candidate test case feature and the target feature, a target test case among the candidate test cases is determined.

[0092] Here, the target features include: a first reference feature of a reference use case, a first requirement feature, and a first code feature.

[0093] For example, the first reference feature includes at least the following factors: the first candidate test case feature, the reference case feature, and the recommended test case feature. The reference case feature is the feature of the reference case obtained when the business change occurs, and the recommended test case feature is the feature of the recommended test case generated by calling the use case recommendation model. Figure 6 , Figure 6 The business change monitoring module 610 triggers the notification mechanism 611 to notify the tester of the business change, obtain the corresponding reference use case, and extract features of the reference use case based on the same principle as the matrix decomposition recommendation model 622 in the use case recommendation model 620 to obtain reference features of the reference use case.

[0094] For example, the similarity can be represented by different types, such as cosine similarity, Euclidean distance, Manhattan distance and correlation coefficient. In the embodiment of the present application, the similarity between two objects is represented by numerical values ​​for illustration.

[0095] For example, after calling the matrix decomposition recommendation model 622 to perform feature extraction, each first candidate test case feature and other features are obtained, the similarity between the first candidate test case feature and the first reference feature, the first requirement feature and the first code feature is calculated, and the similarity matrix between the features obtained according to the similarity calculation is matrix decomposed to determine the target test case among the candidate test cases.

[0096] For example, the matrix decomposition recommendation model 622 is called to perform feature extraction to obtain similarity matrices between each use case, between each requirement, between each code, between use cases and requirements, between use cases and codes, and between projects and codes. Each similarity matrix is ​​decomposed to obtain the corresponding pure number, which is a numerical representation of the similarity between two objects.

[0097] In some embodiments, step 30122 may determine the target test case by at least one of the following sequences:

[0098] Mode 1: when the target feature includes the first requirement feature, determining a first similarity between each first candidate test case feature and the first requirement feature; and sorting the candidate test cases in descending order based on the first similarity to obtain a first sequence.

[0099] Mode 2: When the target feature includes the first code feature, determine the second similarity between each first candidate test case feature and the first code feature, and sort the candidate test cases in descending order based on the second similarity to obtain a second sequence.

[0100] Mode 3: When the target feature includes the first reference feature, determine the third similarity between each first candidate test case feature and the first reference feature, and sort the candidate test cases in descending order based on the third similarity to obtain a third sequence.

[0101] For example, the target test cases are determined based on at least one of the first sequence, the second sequence, and the third sequence. For example, a first preset number of candidate test cases at the head of the first list, the second list, and the third list are used as target test cases.

[0102] For example, the matrix decomposition recommendation model 622 is called to perform a pure numerical calculation on the similarity matrix between the first candidate test case feature and the first requirement feature, represented as Sim2(e, f), which can be implemented by formula (2):

[0103]

[0104] Among them, the first requirement includes: the project's own requirement change document, the requirements of the use case recommendation model in the basic data source, and the overall requirements within the use case recommendation model. e is the pure number of the similarity of the project's own requirement change document, f is the pure number of the similarity of the requirements of the use case recommendation model in the basic data source, g is the pure number of the similarity of the overall requirements within the use case recommendation model, and h is the pure number of the similarity of the first candidate test case.

[0105] The pure numbers of the similarity matrix between the overall project requirements and the candidate test cases, as well as the pure numbers of the similarity matrix between the changed requirements and the requirements in the first corpus are obtained, the mean of the sum of the two pure numbers is taken as the first similarity, and the candidate test cases are sorted in descending order according to the size of the first similarity to obtain a first sequence.

[0106] For example, the matrix decomposition recommendation model 622 is called to perform a pure numerical calculation on the similarity matrix between the first candidate test case feature and the first code feature, represented as Sim3(l, m), which can be implemented by formula (3):

[0107]

[0108] Among them, the first code includes: change code, similar code of the use case recommendation model and internal code of the use case recommendation model, l is the pure number of similarity of the change code, m is the pure number of similarity of the similar code of the use case recommendation model, n is the pure number of similarity of the internal code of the use case recommendation model, and q is the pure number of similarity of the first candidate test case.

[0109] The pure numbers of the similarity matrix between the internal code of the project and the candidate test cases, and the pure numbers of the similarity matrix between the changed code and the code in the first corpus are obtained, and the mean of the sum of the two pure numbers is taken as the second similarity. The candidate test cases are sorted in descending order according to the size of the second similarity to obtain a second sequence.

[0110] For example, the matrix decomposition recommendation model 622 is called to perform a pure numerical calculation on the similarity matrix between the first candidate test case feature and the first reference feature, represented as Sim1(i, j), which can be implemented by formula (1):

[0111]

[0112] Among them, Sim1(i, j) represents the similarity between the first candidate test case i and the first reference feature j, A pure number representing the matrix A(i, j), represents a pure number, and A(i, j) represents a matrix.

[0113] The similarity matrix pure number between the reference case and the candidate test case is obtained as the third similarity, and the candidate test cases are sorted in descending order according to the third similarity to obtain a third sequence.

[0114] For example, descending sorting is to sort the values ​​from large to small according to the matrix pure number of similarity. The first preset number of candidate test cases at the head of the first sequence, the second sequence and the third sequence are used as target test cases. In the embodiment of the present application, the first preset number of heads is 20, that is, the top 20 test cases in the first sequence, the second sequence and the third sequence are used as target test cases.

[0115] Continue to see Figure 3C In step 30123, the target test case and the reference case are used as the first test case.

[0116] For example, the first 20 test cases in the first sequence, the second sequence, and the third sequence are taken as target test cases, and together with the reference test cases, are used as the first test cases generated in the first manner.

[0117] In an embodiment of the present application, the similarity matrix and matrix decomposition can more accurately find candidate test cases related to the target test cases, improve the test coverage, process the use case data from the similarity of multiple dimensions, improve the quality of the test cases generated by the use case recommendation model, perform data enhancement and noise injection after sorting the use cases, and make the test case recommendations more accurate.

[0118] Continue to see Figure 3A , in step 302, a plurality of first test cases are added to a test case database.

[0119] For example, multiple target test cases generated based on the first method and reference test cases are stored in a test case database, and the test case database includes multiple first test cases and test cases in the first corpus database.

[0120] In some embodiments, generating a plurality of first test cases based on the first method is implemented by a test case recommendation model, see Figure 3D , Figure 3D It is the fourth flow chart of the test case generation method provided in the embodiment of the present application; after step 302, the embodiment of the present application also obtains a sample test case, which is implemented through steps 3021 to 3023, as described in detail below.

[0121] In step 3021, a first preset number of candidate test cases are extracted from the heads of the first sequence, the second sequence, and the third sequence, and data enhancement is performed on the extracted first preset number of candidate test cases to obtain a plurality of sample test cases.

[0122] For example, multiple first test cases are generated by calling a test case recommendation model. After the multiple first test cases are generated, the test case recommendation model needs to be optimized based on the generated candidate test cases.

[0123] Continue to see Figure 6 , calling the collaborative filtering recommendation model 623, extracting a first preset number of candidate test cases at the head of the first sequence, the second sequence, and the third sequence. In the embodiment of the present application, the first preset number is 10, that is, the collaborative filtering recommendation model 623 extracts the top 10 candidate test cases in the first sequence, the second sequence, and the third sequence, and performs data enhancement. Data enhancement performs a transformation operation on the candidate test cases to generate new candidate test cases. The transformation operation can be rotation, cropping, scaling, flipping, and adding noise.

[0124] In step 3022, a second preset number of candidate test cases are extracted from the tails of the first sequence, the second sequence, and the third sequence, and noise is injected into the extracted second preset number of candidate test cases to obtain a plurality of sample test cases.

[0125] For example, the collaborative filtering recommendation model 623 is called to extract the second preset number of candidate test cases at the end of the first sequence, the second sequence, and the third sequence. In the embodiment of the present application, the second preset number is 10, that is, the collaborative filtering recommendation model 623 extracts the 10 candidate test cases ranked last in the first sequence, the second sequence, and the third sequence for noise injection. Data noise injection is the process of adding noise to candidate test cases, including: random noise injection, artificial error injection, and abnormal data injection.

[0126] In step 3023, the test case recommendation model is called based on multiple sample test cases for training processing to obtain a trained test case recommendation model.

[0127] Here, the trained test case recommendation model is used to generate multiple first test cases based on the first method.

[0128] For example, after obtaining multiple sample test cases, the adaptive matrix estimation algorithm is called to train the parameters of the test case recommendation model. The adaptive matrix estimation algorithm adjusts its adaptive learning rate according to the different historical gradients of each parameter, adjusts the parameters of the test case recommendation model, and obtains the trained test case recommendation model.

[0129] In an embodiment of the present application, the training data set can be expanded through data enhancement and noise injection, and the generalization ability of the test case recommendation model for test case generation in different situations can be improved. The head and tail of the sequence can be extracted to effectively balance the distribution in the training data set and reduce the risk of overfitting the model to a specific data set. Sample test cases are obtained through a collaborative filtering recommendation model and fed back to the test case recommendation model. The adaptive matrix estimation algorithm can quickly adjust the model parameters to achieve optimization of the model parameters. The quality and efficiency of the recommended cases can be improved by continuously training and updating the test case recommendation model.

[0130] Continue to see Figure 3A In step 303, when the current state of the test case database satisfies the first condition, multiple second test cases are generated based on the second method.

[0131] Here, the second test case is a test case different from the test case generated in the first test case generation phase.

[0132] For example, the first method is continuously executed iteratively, and the generated multiple first test cases are stored in the initial test case database. When the current state of the initial test case database meets the first condition, the first method is switched to the second method, that is, when the state of the test case database generated in the initial stage of the project meets the conditions for executing the mid-term stage of the project, the project can be transferred to the use case generation method for executing the mid-term stage. After the switch, multiple second test cases different from the first test case generation stage are generated based on the second method.

[0133] In some embodiments, determining that the first condition is satisfied in step 303 can be achieved by the following method: when the total number of first test cases in the test case database is greater than or equal to a first threshold, and the ratio of the first number of first test cases with defects in the test case database to the total number of software defects is less than a second threshold, determining that the current state of the test case database satisfies the first condition.

[0134] For example, the first condition of the embodiment of the present application is that the total number of first test cases in the initial test case library is greater than or equal to the first threshold value. The first threshold value of the embodiment of the present application is three thousand, that is, the total number of first test cases is greater than three thousand, and the ratio of the first number of first test cases with defects in the test case database to the total number of software defects is less than the second threshold value. The second threshold value of the embodiment of the present application is 1%, that is, when the defect escape rate is less than 1%, the second mode is switched to be executed. The defect escape rate is the first ratio between the first number and the total number of software defects. It is the ratio of the number of defects still existing in the software product after a specific test phase to the number of all defects known before the same phase, which represents the proportion of defects that were not discovered during the test phase.

[0135] That is, the first condition includes: the total number of first test cases in the test case database is greater than or equal to the first threshold, and the ratio of the first number of first test cases with defects in the test case database to the total number of software defects is less than the second threshold.

[0136] In some embodiments, see Figure 3E , Figure 3E It is the fifth flow chart of the test case generation method provided in the embodiment of the present application; in step 303, generating multiple second test cases based on the second method can be implemented through steps 3031 to 3034, which are described in detail below.

[0137] In step 3031, data enhancement is performed on the second corpus database, and test case recommendation processing is performed based on the data-enhanced second corpus database to obtain a plurality of recommended test cases.

[0138] Here, the second mode is used to instruct to generate a test case based on a second corpus database, where the second corpus database is an updated first corpus database.

[0139] For example, test cases are iteratively generated in the early stage of the project, and the first corpus database is continuously updated based on multiple first test cases generated in the first way. When new test cases are generated, new corpus data are collected from the Internet according to business needs to fill the first corpus database. When the generation method is switched to the second method, it indicates that the use case status update in the first corpus database meets the first condition, and the updated first corpus database is used as the second corpus database for test case generation in the mid-term stage of the project.

[0140] For example, the second corpus database includes different types of keywords, including functions, system characteristics, system data, code flows, and prediction expectations. Data enhancement is performed on the sample sets of test cases based on different types of keywords. Data enhancement is the processing of word segmentation, part-of-speech tagging, and business identification of each test case sample set to increase the number and diversity of test cases in the second corpus database.

[0141] In step 3032, multiple generated test cases are determined based on the updated code data, requirement text, and multiple recommended test cases.

[0142] For example, the continuous integration / continuous delivery tutorial continuously detects changes in the generation phase of the second method. When changes in the second corpus database are detected, a notification mechanism is triggered. Based on the updated code data, requirement text, and multiple recommended test cases, the large language model is called to perform use case generation processing.

[0143] See also Figure 7 , Figure 7 It is a flow chart of the method for generating use cases in the mid-term stage of a project provided in an embodiment of the present application; when the business change monitoring module 710 monitors changes in code data, requirement text, and recommended test cases, the requirement-code-use case large language model 730 is called to generate use cases. The requirement-code-use case large language model 730 is composed of a requirement pre-training model 731, a code pre-training model 732, and a use case pre-training model 733, which are respectively used for pre-training of requirement changes, code data changes, and use case changes.

[0144] For example, multiple recommended test cases are generated by calling the use case recommendation model 720. The principle of generating recommended test cases is the same as the principle of calling the recommended use case generation model in the first method, and will not be repeated here.

[0145] In some embodiments, step 3032 can be implemented by the following method: calling a large language model to perform the following processing: performing text feature extraction processing on the updated code data, requirement text and multiple recommended test cases respectively to obtain a second code feature, a second requirement feature and a recommended case feature of each recommended test case; using the second code feature and the second requirement feature as instructions and the recommended case feature as a reference, iteratively performing character generation processing to obtain a generated test case represented as text.

[0146] Here, the input of each character generation process is the generated characters and the second code feature, the second requirement feature, and the recommended use case feature. For example, the input of the Nth character generation process is all the characters generated by the N-1 character generation processes that have been executed and the second code feature, the second requirement feature, and the recommended use case feature, where N is a positive integer.

[0147] For example, when the code data changes, the changed code repository is matrixed, and the code pre-training model 732 is called to perform feature extraction on the changed code data. Based on the feature values ​​and vectors of the submission records and annotations related to the project code, the feature values ​​of the changes related to the business rules, processes and keywords are obtained, that is, the corresponding second code features are obtained.

[0148] For example, when a requirement text changes, the changed requirement text is matrixed, and the requirement pre-training model 731 is called to perform feature extraction on the updated requirement text. The second requirement feature is obtained based on the business keywords of the project requirement document and the feature values ​​and vectors of the requirement flow chart and the feature values ​​and vectors of the changed requirement document.

[0149] For example, when a recommended test case changes, the case pre-training model 733 is called to perform feature extraction processing on the updated recommended test case, extract the matrix of different business keywords in the recommended test case, and obtain the recommended case features of each recommended test case.

[0150] For example, the second code feature, the second requirement feature and the recommended use case feature are used as hyperparameters, and the large language model is called to iteratively perform character generation processing. The hyperparameters are model parameters in the large language model training process. Through the language generation of the large language model, a generated test case represented as text is obtained.

[0151] Continue to see Figure 3E In step 3033, a reference use case matching the updated business and the updated requirements is obtained.

[0152] For example, when the business and requirements are updated, the notification mechanism of the continuous integration / continuous delivery process is triggered to notify the testers of the updated business and requirements, obtain matching reference use cases, and call the data processing model in the recommended use case model to extract features of the reference use cases to obtain the corresponding reference use case features.

[0153] In step 3034, a plurality of second test cases are generated according to the plurality of recommended test cases, the plurality of generated test cases and the reference cases.

[0154] For example, in the second method, the recommended use case model is called to generate multiple recommended test cases, the large language model is called to generate multiple generated test cases, and the reference cases obtained when the business and requirements change, the three types of use cases are evaluated, and the final use case is determined as the second test case.

[0155] In an embodiment of the present application, the first method of test case generation is dynamically switched to the second method according to the status of the test case database to ensure that the generated test cases match the current software development stage, and the features of code changes, requirement updates and recommended test cases are combined to make the generated test cases more relevant to actual requirements.

[0156] In some embodiments, see Figure 3F , Figure 3F This is the sixth flow chart of the test case generation method provided in an embodiment of the present application; step 3034 can be implemented through steps 30341 to 30345, which are described in detail below.

[0157] In step 30341, the fourth similarity between each recommended test case and the corresponding test requirement is determined, and multiple recommended test cases with the highest fourth similarity are used as target recommended test cases.

[0158] For example, matrix processing is performed between each recommended test case and the test requirement, and the fourth similarity is determined by calculating the cosine similarity between the extracted eigenvalue and the eigenvector, and multiple recommended test cases with the highest fourth similarity are used as target recommended test cases.

[0159] In step 30342, the fifth similarity between each generated test case and the corresponding test requirement is determined, and multiple generated test cases with the highest fifth similarity are used as target generated test cases.

[0160] For example, matrix processing is performed between each generated test case and the test requirement, the cosine similarity between the extracted eigenvalue and the eigenvector is calculated, the fifth similarity is determined, and multiple generated test cases with the highest fifth similarity are used as target generated test cases.

[0161] In step 30343, each reference case is tested to obtain the number of errors in each reference case, and multiple reference cases with the least number of errors are used as target reference cases.

[0162] For example, after determining the reference use case, the reference use case is tested, that is, the reference use case is executed and tested after going online, the number of errors in the reference use case during the detection process is recorded, the defect escape rate of the reference use case is determined based on the number of errors, and the reference use case with the least number of errors is used as the target reference use case.

[0163] In step 30344, each target recommended test case, target generated test case and target reference case are matched, and duplicate parts obtained by matching are removed.

[0164] For example, target recommended test cases, target generated test cases and target reference test cases are matched. There may be repeated text content in the three types of test cases. The repeated text is removed to retain concise and complete test case text content.

[0165] In some embodiments, the plurality of second test cases may also be scored by manually reviewing the plurality of recommended test cases, the plurality of generated test cases, and the reference test cases, and selecting the plurality of test cases with the highest scores.

[0166] In step 30345, multiple second test cases are obtained by splicing in the following manner: at least one part is extracted from the target recommended test cases, target generated test cases and target reference cases after deduplication, and the extracted parts are spliced ​​into a complete second test case.

[0167] For example, the deduplicated test case text is spliced, and at least one part of the target recommended test case, the target generated test case, and the target reference case is extracted and spliced ​​into a complete second test case. The text in the spliced ​​second test case does not contain repeated text content. For example: a second test case includes a test objective, a test object, execution steps, and expected results. The test objectives of the target recommended test case, the target generated test case, and the target reference case are the same. The test objective in the second test case comes from the target recommended test case, the test object comes from the target reference object, and the execution steps and expected results come from the target generated test case.

[0168] In an embodiment of the present application, target recommended test cases, target generated test cases and target reference cases are selected through similarity matching, and the target recommended test cases, target generated test cases and target reference cases are deduplicated. The second test cases obtained by splicing can integrate the advantages of the three types of test cases, thereby improving the coverage and accuracy of generating the second test cases.

[0169] In some embodiments, after step 3032, sample test cases are also obtained, and the large language model is optimized by the sample test cases. Figure 3G , Figure 3G This is the seventh flow chart of the test case generation method provided in the embodiment of the present application; obtaining sample test cases can be achieved through steps 30321 to 30323, which are described in detail below.

[0170] In step 30321, based on the multiple recommended test cases with the highest fourth similarity, the multiple generated test cases with the highest fifth similarity, and the multiple reference cases with the least number of errors, multiple third samples are determined, and data enhancement is performed on the multiple third samples to obtain multiple second positive samples.

[0171] For example, multiple recommended test cases with the fourth highest similarity, multiple generated test cases with the fifth highest similarity, and multiple reference cases with the least number of errors are extracted for data enhancement processing. The principle of extracting test cases is the same as steps 30351 to 30353, and will not be repeated here.

[0172] See also Figure 7 , calling the reward mechanism module 762 in the reinforcement learning model 760 to perform data enhancement on the extracted test cases. The data enhancement performs transformation operations on the test cases, which may be rotation, cropping, scaling, flipping, and adding noise.

[0173] In step 30322, based on multiple recommended test cases with the lowest fourth similarity, multiple generated test cases with the lowest fifth similarity, and multiple reference cases with the highest number of errors, multiple fourth samples are determined, and noise is injected into the multiple fourth samples to obtain multiple second negative samples.

[0174] For example, multiple recommended test cases with the fourth lowest similarity, multiple generated test cases with the fifth lowest similarity, and multiple reference cases with the largest number of errors are extracted for noise injection processing. The principle of extracting test cases is similar to that in step 30321 and will not be repeated here.

[0175] See also Figure 7 , calling the reward mechanism module 762 in the reinforcement learning model 760 to perform noise injection on the extracted test cases. Noise injection is a process of adding noise to training data, including random noise injection, artificial error injection and abnormal data injection. Random noise injection is to inject noise into the test data set by adding random numbers, noise data, etc. Artificial error injection is to artificially create errors in test cases, such as entering incorrect parameters, missing necessary fields, etc., to simulate possible errors in actual applications. Abnormal data injection is to inject abnormal data into the test data set, such as values ​​beyond the normal range, data that does not comply with business rules, etc.

[0176] In step 30323, the model to be trained is trained based on the labeled multiple second positive samples and multiple second negative samples to obtain a large language model, and the large language model is used to perform case generation processing to generate test cases.

[0177] Here, the trained large language model is used to generate multiple second test cases based on the second method.

[0178] For example, each sample test case is reviewed according to the similarity or number of errors corresponding to each sample test case, and the corresponding review scores are obtained. The review scores are sorted, and keywords of the modified business rules and processes are extracted to obtain multiple labeled sample test cases.

[0179] For example, the labeled multiple sample test cases and the extracted business rules and process keywords are taken as hyperparameters, that is, as parameters of the large language model, and are put into the parameter tuner 740. The adaptive matrix estimation algorithm in the parameter tuner 740 is called to adjust the adaptive learning rate of each parameter according to different historical gradients, and the parameters of the large language model are adjusted to obtain the large language model after parameter adjustment.

[0180] In an embodiment of the present application, the large language model can generate corresponding test cases based on business changes. The generated test cases can match the latest updated business rules and stably extract business rules, processes and keywords. The calculation of similarity can accurately feedback the coverage of test cases in the project and improve the synchronization rate of test cases.

[0181] Continue to see Figure 3A In step 304, a plurality of second test cases are stored in a test case database.

[0182] For example, multiple second test cases are iteratively generated based on the second method, the multiple second test cases are stored in a test case database, and the current state of the test case database is continuously updated as a basis for subsequent judgment on whether to switch to the third method; at this time, the test case database includes the first test cases generated in the initial stage and the second test cases generated subsequently.

[0183] In an embodiment of the present application, the second test case is iteratively generated and stored in a test case database, so that test cases can be continuously accumulated to provide a basis for the generation of test cases in subsequent stages. The increase in the number of test cases can improve the test coverage.

[0184] In step 305, when the current state of the test case database satisfies the second condition, test cases of the test case database are generated based on the third method.

[0185] For example, the second method is continuously and iteratively executed, and multiple second test cases generated are stored in the test case database to update the current state of the database. When the switching conditions are met, the second method is switched to the third method. That is, when the state of the use case database generated in the mid-stage of the project meets the conditions for executing the later stage of the project, the project can be transferred to the use case generation method of the post-execution stage. The third method after switching is different from the second method in the second test case generation stage.

[0186] In some embodiments, determining whether the second condition is satisfied in step 305 can be achieved by: determining the first accuracy rate, the first precision rate, and the evaluation value in the current test case database; when the first accuracy rate, the first precision rate, and the evaluation value are all greater than the second ratio threshold, and the ratio between the second number and the total number of software defects is less than the third ratio threshold, determining that the current state of the test case database satisfies the second condition.

[0187] For example, the following parameters corresponding to the current test case database are determined: the total number of recommended test cases in the current test case database; the number of correct test cases of multiple generated test cases in the current test case database; and the second number of test cases whose test results are currently defective.

[0188] For example, judging the current status of the test case database requires judging the accuracy, precision and defect escape rate of the test cases in the test case database. Calculating the accuracy, precision and defect escape rate of the test cases requires determining the total number of test cases recommended by the current test case database, the number of correct test cases of multiple generated test cases and the second number of test cases whose current test results are defective.

[0189] For example, the ratio between the number of correct test cases and the total number of recommended test cases is taken as the first accuracy rate. The first accuracy rate is the ratio between the number of correctly predicted test cases and the total number of test cases in the large language model, and is used to characterize the model accuracy of the large language model.

[0190] For example, a first sum of the number of correct use cases and the second number is determined, and the ratio between the number of correct use cases and the first sum is used as the first precision. In the actual test execution process, there may be a situation where a large language model use case sample is predicted to be correct, but is actually executed as an incorrect sample. The first sum of the number of correct use cases and the second number is determined, and the ratio between the number of correct use cases and the first sum is calculated to obtain the first precision for characterizing the model precision of the large language model.

[0191] For example, the weighted average of the first precision and the first accuracy is determined as the evaluation value. After determining the first precision and the first accuracy of the model, the F1 value of the model is calculated. The F1 value is the weighted average of the precision and the accuracy, which is used to evaluate the balance between the precision and the accuracy of the model. The F1 value can be achieved by formula (4):

[0192] F1=((precision*recall) / (precision+recall))*2 (4)

[0193] Among them, precision is the first accuracy, recall is the first precision, and F1 is the evaluation value between the first accuracy and the first precision.

[0194] For example, when the first accuracy rate, the first precision rate and the evaluation value are all greater than the second ratio threshold, the second ratio threshold of the embodiment of the present application is 90%, that is, when the first accuracy rate, the first precision rate and the evaluation value are all greater than 90%, and the ratio between the second number and the total number of software defects is less than the third ratio threshold, the third ratio threshold of the embodiment of the present application is 0.1%, and the ratio between the second number and the total number of software defects is the defect escape rate of the current test case database, that is, the defect escape rate is less than 0.1%. At the same time, when the review score corresponding to the test case database is higher than 95 points, the second mode is switched to the third mode.

[0195] That is, the second condition includes: when the first accuracy, the first precision and the evaluation value are all greater than the second ratio threshold, and the ratio between the second number and the total number of software defects is less than the third ratio threshold, determining that the current state of the test case database meets the second condition.

[0196] In some embodiments, step 305 obtains test cases from the test case database in the following manner: recommending test cases based on a test case recommendation model to obtain test cases from the test case database; and calling a large language model to perform test case generation processing based on existing test cases in the test case database to obtain new test cases.

[0197] For example, switch to the third method to generate test cases, see Figure 8 , calling the use case recommendation model 820 to recommend test cases, which is the same as the principle of the test case recommendation model 720 in the second mode, and will not be repeated here. Calling the requirement-code-use case large language model 830 to generate test cases, which is the same as the principle of the large language model 720 in the second mode, and will not be repeated here, and the recommended test cases and generated test cases are used as test cases of the third mode.

[0198] In the embodiment of the present application, the judgment basis for switching the project stage is based on the accuracy, evaluation value and precision of the test case, with a certain fault tolerance rate, and the parameter optimization algorithm is called to tune the parameters of the large language model to increase the accuracy and precision of the large language model and ensure the quality of the generated test cases.

[0199] Continue to see Figure 3A In step 306, when the current state of the test case database satisfies the third condition, a step of generating a plurality of first test cases based on the first method is executed.

[0200] For example, the second method is continuously iterated to generate test cases, and a judgment is made as to whether a switching condition is satisfied. When continuous iterations of the second method can satisfy the first condition, the second method of case generation is continued. When any iteration during the generation process of the second method makes the current state of the test case library satisfy the third condition, the second method is switched to the first method, and the step of continuing to generate test cases based on the first method is executed.

[0201] In some embodiments, determining the third condition in step 306 can be achieved in the following manner: determining the first accuracy rate, the first precision rate, and the evaluation value corresponding to the current test case database; when at least one of the first accuracy rate, the first precision rate, and the evaluation value is less than a fourth ratio threshold, or the ratio between the second number and the total number of software defects is greater than the first ratio threshold, determining that the current state of the test case database satisfies the third condition.

[0202] For example, the following parameters corresponding to the current test case database are determined: the total number of recommended test cases in the current test case database; the number of correct test cases of multiple generated test cases in the current test case database; and the second number of test cases whose test results are currently defective.

[0203] Judging the current status of the test case database requires judging the accuracy, precision and defect escape rate of the test cases in the test case database. Calculating the accuracy, precision and defect escape rate of the test cases requires determining the total number of recommended test cases in the current test case database; the number of correct test cases of multiple generated test cases in the current test case database; and the second number of test cases with defects for the current test result.

[0204] For example, the ratio between the number of correct test cases and the total number of recommended test cases is taken as the first accuracy rate. The first accuracy rate is the ratio between the number of correctly predicted test cases and the total number of test cases in the large language model, and is used to characterize the model accuracy of the large language model.

[0205] For example, a first sum of the number of correct use cases and the second number is determined, and the ratio of the number of correct use cases to the first sum is used as the first precision. The first precision is used to characterize the model precision of the large language model, and the calculation principle is the same as in step 305, which will not be repeated here.

[0206] For example, the weighted average of the first precision rate and the first accuracy rate is determined as the evaluation value. The evaluation value of the first precision rate and the first accuracy rate is the F1 value of the model, which is the same as the principle in step 305 and will not be repeated here.

[0207] For example, when at least one of the first accuracy, the first precision and the weighted average is less than the fourth ratio threshold, or the ratio between the second number and the total number of software defects is greater than the first ratio threshold, the second mode is switched to the first mode. In the embodiment of the present application, the fourth ratio threshold is 70%, that is, when at least one of the first accuracy, the first precision and the weighted average is less than 70%, and the ratio between the second number and the total number of software defects is greater than the first ratio threshold, the first ratio threshold of the present application is 1%, that is, when the defect escape rate is greater than 1%, and at the same time, when the review score corresponding to the test case database is less than 70 points, the second mode is switched to the first mode, and the test case generation is continued in the first mode.

[0208] That is, the third switching condition includes: when at least one of the first accuracy, the first precision and the evaluation value is less than the fourth ratio threshold, or the ratio between the second number and the total number of software defects is greater than the first ratio threshold, determining that the current state of the test case database meets the third condition.

[0209] In some embodiments, after step 306, the method further includes: when the current state of the test case database satisfies a fourth condition, switching the third mode to the second mode, and continuing to generate test cases for the test case database based on the second mode; the fourth condition includes: at least one of the first accuracy rate, the first precision rate, and the evaluation value is less than a fifth ratio threshold, or the ratio between the second number and the total number of software defects is greater than a sixth ratio threshold.

[0210] For example, the third method is continuously iterated to generate test cases, and a judgment is made as to whether the switching condition is met. Assuming that the third method can meet the second switching condition after ten consecutive iterations, the use case generation of the third method is continued. When any iteration during the iteration process of the generation of the third method makes the current state of the test case library meet the fourth switching condition, the third method is switched to the second method, and test cases are continued to be generated based on the second method.

[0211] For example, the fourth condition of the embodiment of the present application is that at least one of the first accuracy rate, the first precision rate and the evaluation value is less than the fifth ratio threshold value, the fifth ratio threshold value of the embodiment of the present application is 90%, the review score is lower than 95 points or the ratio between the second number and the total number of software defects is greater than the sixth ratio threshold value, the sixth ratio threshold value of the embodiment of the present application is 0.5%, the currently specified late project use case generation mode is rolled back to the mid-term use case generation mode of the execution project, the third mode is switched to the second mode, and the test case generation is continued in the second mode.

[0212] See also Figure 5 , Figure 5The first method is a test case generation method in the early stage of the project, that is, Figure 5 The first method is the project initial use case generation method 501, and the second method is the test case generation method in the mid-term stage of the project, that is, Figure 5 The third method is the test case generation method in the late stage of the project, that is, Figure 5 How to generate use cases in the later stages of a project 503.

[0213] When the first condition is met, the first mode is switched to the second mode, that is, the initial project use case generation mode 501 is transferred to the mid-term project use case generation mode 502. When the second condition is met, the mid-term project use case generation mode 502 is transferred to the late-term project use case generation mode 503, that is, the second mode is switched to the third mode. When the third condition is met, the mid-term project use case generation mode 502 is rolled back to the initial project use case generation mode 501, that is, the second mode is switched to the first mode. When the fourth condition is met, the late-term project use case generation mode 503 is rolled back to the mid-term project use case generation mode 502, that is, the third mode is switched to the second mode.

[0214] In an embodiment of the present application, in the initial use case generation stage of the project, multiple first test cases are generated based on the first method and added to the test case database. When the conditions are met, the second generation method and the third generation method are switched to determine the currently applicable test case generation method according to the current state of the test case database, which is applicable to generating test cases and executing test cases to test the application in various stages. The generated test cases are evaluated and ranked, and the use case recommendation model and the large language model are fed back based on the top-ranked test cases. The parameters of the model are optimized through a reward mechanism, the performance of the model is enhanced, and the quality of the use cases recommended by the model is improved. The generation method is switched according to the different states of the test case database, so that the test cases match the current needs of the test case library, and the quality and accuracy of the generated test cases are improved.

[0215] The following is an explanation of an exemplary application of the test case generation method provided in an embodiment of the present application in an actual application scenario.

[0216] Test cases are the most important part of software testing, and are used to measure the coverage of software requirements and code paths during software testing. In actual application scenarios, business requirements are cumbersome, and the coverage of related technologies through manually written test cases is low. There is a high degree of reliance on the experience of testers, and the popularity of testers will also lead to a decline in business quality. For new business projects, the initial stage of the project draws on other businesses and relies on the pure manual work of testers to improve the coverage. The cost of manual communication, the cost of human maintenance, and the time cost are very large, the output rate cannot meet high standards, and the coverage of requirements and codes cannot be guaranteed, resulting in repeated defects.

[0217] The relevant technology obtains the template test requirements of the target business, obtains the knowledge of the target business from the target knowledge base, and constructs a demand analysis to generate test cases. Due to the iterative updates of business and software, the relevant technology cannot optimize the models and use cases in a timely and accurate manner, and cannot meet the needs of complex use cases with many operation combinations.

[0218] The test case generation method provided in the embodiment of the present application monitors each stage of project execution through the continuous integration / continuous delivery process, notifies business changes in a timely manner, and executes different use case generation methods for different stages of the project to ensure that the generated test cases are always consistent with the latest business rules and code changes. Through the evaluation rules of different stages, the stage of project execution and rollback is judged, and the accuracy and quality of generated test cases are improved based on the optimization of parameters of different models.

[0219] The following is a description with reference to the accompanying drawings. Figure 4 , Figure 4 The eighth flow chart of the test case generation method provided in the embodiment of the present application, the execution subject can be a terminal device, a server or a combination of the two. The embodiment of the present application takes the server as the execution subject as an example. Figure 4 The steps shown are explained in detail.

[0220] In step 401, business change monitoring and message notification are integrated into the continuous integration / continuous delivery process.

[0221] For example, step 401 is continuously executed and is applicable to each stage of the project.

[0222] For example, the continuous integration / continuous delivery process will perform real-time detection on each stage of the project in the embodiment of the present application, integrate the monitoring and notification of business changes into the continuous integration / continuous delivery process, and when the business requirements or code changes, corresponding message notifications will be made to the different stages of the project. Continuous integration / continuous delivery is a business model that monitors business changes in the development process in real time, and reminds relevant personnel through message notifications when changes occur. In the embodiment of the present application, projects executed at different stages represent software development projects.

[0223] In some embodiments, the continuous integration / continuous delivery process in step 401 is integrated into the overall process of the project. For a more detailed description of the overall execution process of the test case generation method for the project stage, see Figure 5 , Figure 5 The overall process diagram of the project use case generation method provided by the embodiment of the present application is described in detail below. The overall use case generation stage of the project includes: the project initial use case generation method 501, the project mid-term use case generation method 502, and the project late use case generation method 503.

[0224] After the execution of the project initial use case generation method 501 is completed, when the evaluation criteria of the generated use case meet the conditions for executing the mid-term of the project, the execution of the project mid-term use case generation method 502 is transferred. The condition for executing the mid-term stage of the project in the embodiment of the present application is that the accumulation of the project-related resource library reaches 3,000 (it can be set to other numbers in actual applications) iterative sample set data model construction requirements, and the recommended defect escape rate is less than 1%. The defect escape rate is the ratio of the number of defects still existing in the software product after a specific testing phase to the number of all defects known before the same phase.

[0225] In the process of iteratively executing the mid-term use case generation method 502, when the evaluation criteria of the generated use case meet the conditions for the late stage of the project execution, the execution of the project is transferred to the late stage; when the evaluation criteria for continuing the mid-term preset value of the project are not met, the initial use case generation method of the project is rolled back 501. In the embodiment of the present application, the conditions for executing the late stage of the project are that the accuracy rate, F1 value and precision evaluation results are greater than 90%, the defect escape rate is less than 0.1%, and the review score is higher than 95 points; the evaluation criteria for not meeting the mid-term preset value of the project execution are that the accuracy rate, F1 value and precision evaluation results are less than 70%, the review score is lower than 70 points or the defect escape rate is higher than 1%.

[0226] During the iterative execution of the project late use case generation method 503, when the evaluation criteria of the generated use case continue to execute the evaluation criteria of the project late preset value, the project mid-term use case generation method 502 is rolled back. In the embodiment of the present application, the evaluation criteria that do not meet the continued execution of the project late preset value are that at least one of the accuracy, F1 value and precision evaluation results is lower than 90%, the defect escape rate is greater than 0.5%, and the expert review score is lower than 95 points.

[0227] Continue to see Figure 4 In step 402, the use case generation method in the early stage of the project is executed.

[0228] For example, when the project is in its early stages, the amount of data in the basic requirement documents, codes, and use cases cannot meet the demand. At this time, the use case generation mechanism for the early stage of the project (the first method mentioned above) is executed to supplement the use cases.

[0229] In some embodiments, the method for generating use cases in the early stage of the project performed in step 402 is: Figure 5 How to generate use cases in the early stages of a project 501, see Figure 6 , Figure 6 This is a flowchart of a method for generating use cases in the early stage of a project provided in an embodiment of the present application. Step 402 can be implemented through steps 601 to 605, which are described in detail below.

[0230] In step 601, a continuous integration / continuous delivery process is obtained.

[0231] For example, in the initial use case generation stage of the project, the continuous integration / continuous delivery process monitors business changes and sends message notifications when business changes occur. The continuous integration / continuous delivery process calls the business change monitoring module 610. It ensures that each time a change occurs, the corresponding testers can be notified in time and all requirement documents can be obtained in time.

[0232] For example, in the early stages of a project, the amount of data in the basic requirement documents, codes, and use cases cannot meet the needs of the use case recommendation model 620. The use case recommendation model 620 is the test case recommendation model mentioned above. It collects the open source use case libraries, requirement libraries, and code libraries across the company and the entire network as the project resource library 630, and uses the use case recommendation model 620 to supplement the amount of use case data in the project resource library 630.

[0233] In step 602, an artificial use case is obtained based on the business change notification.

[0234] For example, the continuous integration / continuous delivery process continuously monitors whether the business has changed. When the business change monitoring module detects changes in code documents and business requirements, it triggers the notification mechanism of the continuous integration / continuous delivery process to notify the tester of the changes. After receiving the notification of the change, the tester generates an artificial use case corresponding to the change.

[0235] For example, the artificial use cases generated by the testers are stored in the project resource library to supplement the data items of the project resource library.

[0236] In step 603, the use case recommendation model is called to generate recommended use cases.

[0237] For example, when code documentation or business requirements change, the use case recommendation model is called according to the basic project resource library to automatically generate recommended use cases.

[0238] Continue to see Figure 6 When the business change monitoring module 610 monitors changes in code documents or business requirements, it calls the use case recommendation model 620 to execute the use case recommendation process. The use case recommendation model 620 is composed of a data processing model 621, a matrix decomposition recommendation model 622 and a collaborative recommendation filtering model 623.

[0239] When the business change monitoring module 610 monitors a change, the data processing model 621 is called to pre-process the basic data source 630 in the project resource library, including: data deduplication, standardization, and data set segmentation and analysis. Data deduplication is to delete duplicate data items in the data set and reduce redundant information in the data set. Standardization is to unify and normalize the data in the data set to eliminate differences between different data types and units. Data set segmentation is the process of dividing the original data set into a training set, a validation set, and a test set. The pre-processing of the basic data source is completed through the processing of data deduplication, standardization, and data set segmentation and analysis.

[0240] After completing the preprocessing of the basic data source, the matrix decomposition recommendation model 622 is called to perform matrix processing on the project business keywords, business processes, test processes, test expected results, code execution processes and code annotation-related business functions in the basic data source, and extract eigenvalues ​​and corresponding eigenvectors. Using cosine similarity, the similarity between each use case, between each requirement, between each code, between use cases and requirements, between use cases and codes, and between projects and codes is decomposed and matrixed. The resource matrix is ​​decomposed for each similarity matrix obtained by the above matrix processing, and each matrix is ​​decomposed into corresponding pure numbers. Pure numbers are to decompose a matrix into multiple sub-matrices so that the elements contained in the sub-matrices are all numerical values.

[0241] For example, the similarity matrix generates automatic recommendation use cases after resource matrix decomposition, and the automatic recommendation use cases are stored in the project resource library as a supplement to the use case data volume.

[0242] For example, all requirements and codes in the project resource library 630 and the manual test cases obtained in step 602 are matrixed to obtain similarity matrices between each use case, between each requirement, between each code, between use cases and requirements, between use cases and codes, and between projects and codes, and the above similarity matrices are decomposed to obtain the corresponding pure numbers.

[0243] Based on the pure number recommendation method, the similarity between artificial use case i and recommended use case j is calculated as Sim1(i, j), the similarity between the project's own requirement e and the requirement f in the use case recommendation model is calculated as Sim2(e, f), and the similarity between the project's own code change l and the code m in the use case recommendation model is calculated as (l, m). Pure numbers are numerical representations of the similarity between two objects.

[0244] The similarity between use cases is represented as Sim1(i, j), which can be achieved by formula (1):

[0245]

[0246] Among them, Sim1(i, j) represents the similarity between the manual test case i and the automatic recommendation case j of the recommendation model. A pure number representing the matrix A(i, j), represents a pure number, and A(i, j) represents a matrix.

[0247] The similarity between requirements is represented as Sim2(e, f), which can be achieved by formula (2):

[0248]

[0249] Among them, e is the pure number of requirement change documents itself, f is the pure number of requirements for the recommended model in the basic data source, g is the pure number of internal projects for the overall requirements of the use case recommendation model, and h is the pure number of internal use cases of the use case recommendation model.

[0250] The similarity between code changes is represented as Sim3(l, m), which can be achieved by formula (3):

[0251]

[0252] Among them, l is the pure number of changed codes, m is the pure number of similar codes in the use case recommendation model, n is the pure number of codes inside the use case recommendation model, and q is the pure number of use cases inside the use case recommendation model.

[0253] Continue to see Figure 6, in step 604, a use case review is performed to obtain a final use case.

[0254] For example, the final use case in the early stage of the project is the first test case mentioned above. The matching degree is ranked according to the similarity of the three dimensions, and the automatically recommended use cases are manually ranked and scored according to business keywords, business processes, test processes, and expected test results.

[0255] For example, similarity matching is performed based on the keywords and processes of the project in combination with automatic recommended use cases, and expert review is performed based on the three-dimensional similarity calculated by the purity recommendation algorithm and the coverage. In the embodiment of the present application, the three dimensions are sorted separately, and the top 20 use cases are taken out for review. The more scenarios the recommended use cases cover, the higher the score, and the corresponding keywords are stored in the project requirements library.

[0256] For example, based on the ranking of automatically recommended test cases, manual test cases and reviews, the manual test cases are standardized, the test case coverage is adjusted, and the final test cases are obtained.

[0257] In step 605, the final use case is executed.

[0258] For example, the final use case is executed during the test process and the online process, and the defect escape rate of the use case is calculated based on the online escape of the use case after the test process execution and each online release. The defect escape rate of the use case indicates the proportion of defects that were not discovered during the test phase.

[0259] For example, the calculated use case defect escape rates are ranked, and combined with the expert review rankings, the top 10 use cases are data augmented, and noise injection is performed on the bottom 10 use cases. Noise injection is a process of adding noise to training data. Through artificial error injection, errors are artificially created in test cases, such as entering incorrect parameters, missing necessary fields, etc., to simulate possible errors in actual applications.

[0260] Continue to see Figure 6 , extract the matching degree of the automatically recommended use cases, the business rules and process keywords modified in the manual use cases and the final use cases, call the collaborative filtering recommendation model 623, optimize the parameters of the matrix decomposition recommendation model 622 and the use case recommendation model 620 based on the adaptive matrix estimation algorithm, and adjust its adaptive learning rate according to the different historical gradients of each parameter, and feed back the top 10 use cases to the basic data source. The use case recommendation model 620 with optimized parameters iteratively executes the initial use case generation method of the project to supplement the data volume of the project resource library.

[0261] Continue to refer Figure 4In step 403, when the evaluation criteria for executing the mid-term preset value of the project are met, the mid-term stage use case generation method of the project is executed.

[0262] For example, after the use cases of the basic database are supplemented and optimized in the early stage of the project, the continuous integration / continuous delivery process monitors that the current project meets the conditions for executing the mid-term stage of the project (the first condition mentioned above), and the project execution is transferred from the early stage to the use case generation method for executing the mid-term stage of the project (the second method mentioned above).

[0263] For example, the conditions preset in the embodiment of the present application for meeting the mid-term stage of executing the project are that the accumulation of project requirement library, code library and use case library reaches three thousand iterative sample set data, and the use case library collected from the entire network open source, company and project reaches one million, and the defect escape rate after execution of the current recommended use case is less than 1%. At this time, the project execution can be transferred from the initial stage to the mid-term stage, and the method of generating test cases is also switched to a method suitable for the mid-term stage.

[0264] In some embodiments, the method of generating use cases in the mid-term phase of the project performed in step 403 is similar to Figure 5 The use cases in the early stage of the 502 project are generated in the same way, see Figure 7 , Figure 7 It is a flowchart of the method for generating use cases in the mid-term stage of a project provided in an embodiment of the present application; the method for generating use cases in the mid-term stage of a project executed in step 403 is implemented through steps 701 to 707, which are described in detail below.

[0265] In step 701, a continuous integration / continuous delivery process is obtained.

[0266] For example, the continuous integration / continuous delivery process continues to monitor business changes in the mid-term stage of the project, and the continuous integration / continuous delivery process calls the business change monitoring module 710. The tester is notified every time a change occurs, and after the continuous integration / continuous delivery process is completed, the code and requirement documents of the branch changes are automatically pulled.

[0267] In step 702, an artificial use case is obtained based on the business change notification.

[0268] For example, in the mid-term stage of the project, the continuous integration / continuous delivery process continues to monitor whether the business has changed. When the business change monitoring module 710 detects changes in code documents and business requirements, it triggers the notification mechanism 711 to notify the tester of the changes. After receiving the notification of the changes, the tester generates an artificial use case corresponding to the changes.

[0269] The use case recommendation model in the mid-term stage of the project has the same composition as the use case recommendation model in the initial stage of the project, and the principle of executing the use case recommendation process is the same. The use case recommendation model 720 is composed of a data processing model, a matrix decomposition recommendation model, and a collaborative recommendation filtering model. After the artificial test case is generated, the matrix decomposition recommendation model in the use case recommendation model 720 is called to perform matrix processing on the artificial test case to extract the eigenvalues ​​and eigenvectors of the artificial test case.

[0270] In step 703, the use case recommendation model is called to generate recommended use cases.

[0271] For example, when the business change monitoring module 710 monitors changes in code documents and business requirements, the notification mechanism 711 is triggered, and the use case recommendation model 720 is called based on the business change notification to automatically recommend use cases. When the business change monitoring module 710 monitors changes, the use case recommendation model 720 is called to generate recommended use cases. The process of automatically recommending use cases is the same as the principle of the initial step 603 of the project, which will not be repeated here.

[0272] In step 704, a project validation set is constructed.

[0273] For example, obtain the iterative sample set generated in the early stage of the project, perform data enhancement on the project requirement library, project use case library and project code library in the iterative sample set, and perform data enhancement on the samples in the original data set to generate new samples. The transformation operation can be rotation, cropping, scaling, flipping, and adding noise.

[0274] Collect keywords such as functions, system characteristics, system data, code flow, test expectations, etc. of the iterative sample set, and perform word segmentation, part-of-speech tagging, and business identification on each sample set. Word segmentation is to divide the data in the sample set according to certain rules and standards, and divide it into different words or tags. Part-of-speech tagging is to classify each word in the text and mark it according to different parts of speech. Perform the same data enhancement processing on the recommended use cases in step 703, input the automatic recommended use cases and iterative sample sets after data enhancement processing into the same matrix decomposition model as in the use case recommendation model 720, perform matrix processing, extract the eigenvalues ​​and eigenvectors of the corresponding use cases, and construct a project verification set by calculating the similarity.

[0275] In step 705, the requirement-code-use case large language model is called to automatically generate test cases.

[0276] For example, based on the millions of data of all requirements, codes and use cases collected at the beginning of the project, pre-training is performed to generate a large language model of requirements-code-use cases, and the model parameters of the large language model of requirements-code-use cases are tuned using the parameters at the beginning of the project. When the requirements document and code are changed in the business monitoring module, the large language model of requirements-code-use cases obtains the changes and automatically generates test cases.

[0277] For example, we call the Bidirectional and Auto-Regressive Transformer model to pre-train on the data collected at the beginning of the project. The Bidirectional and Auto-Regressive Transformer model is a model that combines the properties of bidirectional and autoregressive, generating a sequence of output tokens one at a time, conditioned on previously generated tokens and the input provided by the encoder, and pre-trained on a large corpus.

[0278] The data preprocessed by calling the bidirectional autoregressive transformer model include requirements, codes and use cases, and a requirement pre-training model 731, a code pre-training model 732 and a use case pre-training model 733 are respectively constructed to form a requirement-code-use case large language model 730.

[0279] When a requirement document is changed, the requirement pre-training model 731 performs matrix processing on the changed requirement document, extracts the business keywords, eigenvalues ​​and eigenvectors of the project requirement document, and the eigenvalues ​​and eigenvectors of the requirement flow chart, calculates the similarity between the corresponding eigenvalues ​​and eigenvectors, and performs data enhancement on similar eigenvalues ​​and eigenvectors.

[0280] When a code change occurs, the code pre-training model 732 performs matrix processing on the changed code repository, extracts the eigenvalues ​​and eigenvectors of the submission records and annotations related to the project code, obtains the eigenvalues ​​of changes related to business rules, processes, and keywords, calculates the similarity between the eigenvalues ​​and eigenvectors, and performs data enhancement on similar eigenvalues ​​and vectors.

[0281] The millions of data collected at the beginning of the project include recommended use cases generated by the use case recommendation model at the beginning of the project. The use case pre-training model 733 extracts the eigenvalues ​​and eigenvectors of the recommended use cases generated at the beginning of the project as hyperparameters, which are fed back to the requirement pre-training model 731. Hyperparameters are parameters that need to be manually set during the model training process. The similarity association matrix is ​​transmitted between the three models of the requirement pre-training model 731, the code pre-training model 732 and the use case pre-training model 733. The similarity association matrix is ​​a matrix obtained by calculating the similarity after the eigenvalues ​​and eigenvectors summarized by the pre-training model are matrixed. The recommended use case calls the requirement-code-use case large language model 730 for language generation and automatically generates test cases. Language generation is to extract or construct meaningful language expressions or outputs from data or information through specific algorithms or models.

[0282] In step 706, a use case set is constructed.

[0283] For example, collect automatically generated test cases generated by the requirement-code-use case large language model when business changes occur, call the recommended use cases and manual use cases generated by the use case recommendation model, and build a use case set.

[0284] In step 707, the use case set and the manual use case coverage are reviewed to obtain the final use case.

[0285] For example, an expert review is conducted on the use case set consisting of automatically generated test cases, recommended use cases and artificial use cases and the coverage of artificial use cases, the eigenvalues ​​and eigenvectors of artificial use cases are extracted, the cosine similarity with the project keyword matrix in the project verification set is calculated, and the coverage of artificial use cases is determined.

[0286] The scores obtained from the automatically generated test cases, recommended test cases, manual test cases, and the expert review of the coverage of manual test cases are sorted, and the keywords of the modified business rules and processes are extracted. The parameters of the use case recommendation model 720, the parameters of the requirements-code-use case large language model 730, and the keywords of the modified business rules and processes are input into the parameter tuner 740. The parameter tuner 740 tunes each parameter through the Adaptive Moment Estimation (Adam) algorithm. The Adaptive Matrix Estimation algorithm is a commonly used optimization algorithm used for the training of neural network models. It combines the concepts of gradient descent and momentum and can update model parameters more effectively.

[0287] For example, based on artificial use cases, recommended use cases, automatically generated test cases, artificial use case coverage and expert review scores, duplicate information in the three types of use cases is deleted, and the final use cases are determined through an overall evaluation.

[0288] In step 708, the final use case is executed.

[0289] For example, after the final use case is determined through evaluation, the final use case is executed and tested in two scenarios: the test environment and the project launch. The defects found in the final use case during the test process and the execution process after the launch are recorded, the ratio of the two defects is calculated, the use case defect escape rate is determined, and the ranking of the automatically generated test cases is determined based on the use case defect escape rate.

[0290] For example, based on the review results of three use cases and the determined use case defect escape rates, the evaluation model 750 is called to calculate the accuracy, precision and F1 value of the requirement-code-use case large language model 730 .

[0291] The accuracy rate (the first accuracy rate mentioned above) is the ratio of the number of correctly predicted samples to the total number of use case samples in the automatically generated test cases generated by calling the requirement-code-use case large language model 730, which is denoted as precision. That is, the accuracy rate of the requirement-code-use case large language model 730 is equal to the ratio of the use cases left by the review to the total generated use cases.

[0292] The accuracy rate (the first accuracy rate mentioned above) is the ratio of the number of samples predicted correctly to the number of samples actually predicted correctly in the automatically generated test cases generated by calling the requirement-code-use case large language model 730, which is denoted as recall. Due to the defects in the model's prediction, there may be samples predicted as correct by the model, but found to be negative error samples during the actual use case execution. That is, the accuracy rate of the requirement-code-use case large language model 730 is equal to the ratio between the use cases correctly executed in the test and the total use cases left by the review.

[0293] The F1 value (the above evaluation value) is determined by the accuracy and precision. The F1 value can be achieved by formula (4):

[0294] F1=((precision*recall) / (precision+recall))*2 (4)

[0295] Among them, precision is the accuracy of the 740 requirement-code-use case large language model, recall is the precision of the 730 requirement-code-use case large language model, and F1 is the weighted average between accuracy and precision.

[0296] For example, if the calculated accuracy, F1 value and precision evaluation results are less than 70%, the use case score is lower than 70 points or the defect escape rate is higher than 1%, the performance of the use case recommendation model 720 is improved, and according to the evaluation results, the expert review modification parameters and the parameters of the use case recommendation model 720 are adjusted. At the same time, the project is returned to the initial use case generation stage, and the method of generating test cases is also switched to a method suitable for the initial stage, increasing the sample sets of project requirements, codes and use cases, and improving the accuracy of the use case recommendation model 720.

[0297] Feedback the evaluation indicators in the evaluation model 770: accuracy, F1 value and precision, and the top 10 use cases in step 707 to the reinforcement learning model 760, input the top 10 use cases into the reward mechanism module 762, perform data enhancement on the top 10 use cases, and inject data noise into the bottom 10 use cases. Data noise injection is a process of adding noise to training data, including random noise injection, artificial error injection, and abnormal data injection. Input the accuracy, F1 value and precision into the parameter adjustment module 761, and use the keywords of the business rules and processes modified by the expert review to tune the parameters of the requirement-code-use case large language model 730.

[0298] For example, if the calculated accuracy, F1 value and precision evaluation results are greater than 90%, the defect escape rate is less than 0.1%, and the expert review score is higher than 95 points, then the iterative evaluation standard of the use case generation method in the mid-term project meets the requirements of the later stage of the project, and the use case generation method in the later stage of the project is switched to, and the method of generating test cases is also switched to a method suitable for the later stage.

[0299] Continue to refer Figure 4 In step 404, when the evaluation criteria for executing the preset value of the later stage of the project are met, the use case generation method for the later stage of the project is executed.

[0300] For example, when the evaluation criteria for the preset values ​​of the later stage of the execution project (the second condition mentioned above) are met, that is, during the execution of the mid-term stage of the project, the calculated accuracy, F1 value and precision evaluation results are greater than 90%, the defect escape rate is less than 0.1%, and the expert review score is higher than 95 points, the use case generation method of the later stage of the project is executed (the third method mentioned above).

[0301] In some embodiments, in some embodiments, the project later stage use case generation method performed in step 404 is similar to Figure 5 The use cases in the early stage of the 503 project are generated in the same way, see Figure 8 , Figure 8It is a flowchart of the method for generating use cases in the later stage of a project provided in an embodiment of the present application; the method for generating use cases in the later stage of a project executed in step 404 is implemented through steps 801 to 807, which are described in detail below.

[0302] In step 801, a continuous integration / continuous delivery process is obtained.

[0303] For example, the continuous integration / continuous delivery process continues to monitor business changes in the mid-stage of the project, notifies the use case recommendation model every time a change occurs, and automatically pulls the code and requirement documents of the branch changes after the continuous integration / continuous delivery process is completed. In the use case generation method in the later stage of the project, there is no need to notify testers to generate manual use cases when business changes occur.

[0304] In step 802, the use case recommendation model is called to generate recommended use cases.

[0305] For example, when the business change monitoring module 810 monitors changes in code documents and business requirements, the notification mechanism 811 is triggered, and the use case recommendation model 820 is called based on the business change notification to automatically recommend use cases. When the business change monitoring module 710 monitors changes, the use case recommendation model 820 is called to generate recommended use cases. The process of automatically recommending use cases is the same as the principle of step 603 in the initial stage of the project, and will not be repeated here.

[0306] In step 803, a project validation set is constructed.

[0307] For example, obtain the iterative sample set generated in the mid-term stage of the project, perform data enhancement processing on the project requirement library, project use case library and project code library in the iterative sample set, and perform data enhancement on the samples in the original data set to generate new samples. The transformation operation can be rotation, cropping, scaling, flipping, and adding noise. After performing the data enhancement operation, call the matrix decomposition model to matrix the data to obtain the project verification set. The line of sight principle of the matrix decomposition model is the same as the matrix decomposition model principle in the use case recommendation model 720, which will not be repeated here.

[0308] In step 804, the requirement-code-use case large language model is called to automatically generate test cases.

[0309] For example, when the requirement document and code are changed, the continuous integration / continuous delivery process obtains the changes, calls the requirement-code-use case large language model 830, and executes the process of automatically generating test cases.

[0310] For example, the requirement-code-use case large language model 830 is composed of a requirement pre-trained model 831, a code pre-trained model 832 and a use case pre-trained model 833. The principle of automatically generating use cases is the same as the principle of the requirement-code-use case large language model 730 in the use case generation method in the mid-term project stage, and will not be repeated here.

[0311] In step 805, a use case set is constructed.

[0312] For example, the automatically generated test cases generated by the requirement-code-use case large language model are collected when the business changes, and the recommended use cases generated by the use case recommendation model 820 are called to build a use case set. Different from the use case generation method in the mid-stage of the project, the use case generation method in the late stage of the project does not include manual use cases in the use case set.

[0313] In step 806, the review scores of the automatically generated test cases and the recommended cases are determined to obtain the final cases.

[0314] For example, an expert review is conducted on a use case set consisting of automatically generated test cases and recommended use cases, and the review scores are sorted to extract the keywords of the modified business rules and processes. The parameter tuner 840 is called to tune the parameters of the use case recommendation model 820, the parameters of the requirement-code-use case large language model 830, and the keywords of the modified business rules and processes. The principle of the parameter tuner 840 is the same as that of the parameter tuner 740 in the use case generation method in the mid-term stage of the project, and will not be repeated here.

[0315] For example, based on the recommended use cases, automatically generated test cases and expert review scores, duplicate information in the use cases is deleted, and the final use cases are determined through an overall evaluation.

[0316] In step 807, the final use case is executed.

[0317] For example, after the final use case is determined through evaluation, the final use case is executed and tested in two scenarios: the test environment and the project launch. The defects found in the final use case during the test process and the execution process after the launch are recorded, the ratio of the two defects is calculated, the use case defect escape rate is determined, and the ranking of the automatically generated test cases is determined based on the use case defect escape rate.

[0318] For example, based on the review results of the two use cases and the determined use case defect escape rate, the evaluation model 850 is called to calculate the accuracy, precision and F1 value of the requirement-code-use case large language model 830. The calculation method of the accuracy, precision and F1 is the same as that in the use case generation method in the mid-term stage of the project, and will not be repeated here.

[0319] The evaluation indicators in the evaluation model 850: accuracy, F1 value and precision, and the top 10 use cases in step 806 are fed back to the reinforcement learning model 860, and the parameters of the requirement-code-use case large language model 830 are tuned. The principle of parameter tuning is the same as the tuning principle of the reinforcement learning model 760 in the use case generation method in the mid-term stage of the project, and will not be repeated here.

[0320] For example, in the later stage of the project, the use case generation method continues to iterate. After 10 consecutive iterations, the five dimensional indicators of accuracy, F1 value and precision evaluation results greater than 90%, defect escape rate less than 0.1%, and expert review score higher than 95 points can be met. This proves that the requirement-code-use case large language model and the use case recommendation model can meet the daily changes in project quality and can fully utilize the model to complete use case generation. At this time, the project later generation mechanism can continue to be used to continue to generate use cases.

[0321] For example, in the process of continuous iteration of use case generation in the later stage of the project, if the accuracy, F1 value and precision evaluation results are lower than 90%, the defect escape rate is greater than 0.5%, and the expert review score is lower than 95 points (the third condition above), it is necessary to roll back to the project mid-term generation mechanism to generate use cases, and the method of generating test cases is also switched to a method suitable for the mid-term stage. Data enhancement is performed on use cases with high indicator rankings, noise is injected into use cases with low indicator rankings, and the requirement-code-use case large language model and use case recommendation model are optimized.

[0322] The test case generation method provided in the embodiment of the present application has the following beneficial effects:

[0323] By monitoring each stage of project execution through the continuous integration / continuous delivery process, relevant testers and related use case generation models can be notified in a timely manner when business changes occur, and use case evaluations can be performed on the correlation between recommended use cases and artificial use cases and internal ones, and use cases with reference value can be recommended. The accuracy of recommended use cases can be improved by using manual review sorting and data enhancement and noise injection on recommended data. When business rules change, the solution that calls the code-requirement-use case model to generate test cases will generate test cases that meet the iteration in real time, ensuring that the generated test cases are always consistent with the latest business rules and code changes. Through the evaluation rules of different stages, the stages of project execution and rollback are judged, and based on the optimization of parameters of different models, stable use case generation is achieved to ensure the high-quality and uninterrupted progress of the project, saving more labor costs.

[0324] The following is a description of an exemplary structure of the test case generating device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2As shown, the software modules stored in the test case generating device 455 of the memory 450 may include: a first case generating module 4551, used to generate multiple first test cases based on a first method; add the multiple first test cases to a test case database; a second case generating module 4552, used to generate multiple second test cases based on a second method when the current state of the initial test case database satisfies the first condition; store the multiple second test cases in the test case database; a third case generating module 4553, used to generate test cases for the test case database based on a third method when the current state of the test case database satisfies the second condition; and execute the step of generating multiple first test cases based on the first method when the current state of the test case database satisfies the third condition.

[0325] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the test case generation method described in the embodiment of the present application.

[0326] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the test case generation method provided by the present application embodiment, for example, Figure 3A The test case generation method shown.

[0327] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0328] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0329] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0330] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0331] To sum up, the embodiment of the present application determines the currently applicable test case generation method according to the current state of the test case database, which is suitable for generating test cases and executing test cases in various stages of testing the application, so that the test cases match the current needs of the test case library, thereby improving the quality and accuracy of the generated test cases.

[0332] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A test case generation method, characterized in that: The method comprises: Generate multiple first test cases based on the first method; adding the plurality of first test cases to a test case database; When the current state of the test case database satisfies the first condition, generating a plurality of second test cases based on a second method; Storing the plurality of second test cases in the test case database; When the current state of the test case database satisfies the second condition, generating test cases for the test case database based on a third method; When the current state of the test case database satisfies the third condition, the step of generating a plurality of first test cases based on the first method is performed.

2. The method according to claim 1, characterized in that The first mode is used to instruct to generate a test case based on the first corpus database; The generating of a plurality of first test cases based on the first method comprises: Performing data cleaning on the first corpus database to obtain a basic database; The plurality of first test cases are generated based on the basic database.

3. The method according to claim 2, characterized in that The first corpus database includes the following types of data: candidate test cases, requirement texts, and code data of the application to be tested; The generating the plurality of first test cases based on the basic database comprises: Performing feature extraction processing on each type of data in the basic database to obtain a first candidate test case feature of each candidate test case, a first requirement feature of each requirement text, and a first code feature of each code data; Determine a target test case among the candidate test cases based on the similarity between each of the first candidate test case features and a target feature, wherein the target feature includes at least one of the following: a first reference feature of a reference case, the first requirement feature, and the first code feature; The target test case and the reference test case are used as the first test case.

4. The method according to claim 3, characterized in that The target feature includes at least one of the following: the first reference feature, the first requirement feature, and the first code feature; The determining, based on the similarity between each of the first candidate test case features and the target feature, a target test case among the candidate test cases comprises: In the case where the target feature includes the first requirement feature, determining a first similarity between each of the first candidate test case features and the first requirement feature; and sorting the candidate test cases in descending order based on the first similarity to obtain a first sequence; In the case where the target feature includes the first code feature, determining a second similarity between each of the first candidate test case features and the first code feature; and sorting the candidate test cases in descending order based on the second similarity to obtain a second sequence; In the case where the target feature includes the first reference feature, determining a third similarity between each of the first candidate test case features and the first reference feature; and sorting the candidate test cases in descending order based on the third similarity to obtain a third sequence; The target test case is determined based on at least one of the first sequence, the second sequence, and the third sequence.

5. The method according to claim 1, characterized in that The method further comprises: When the total number of first test cases in the test case database is greater than or equal to a first threshold, and the ratio of the first number of first test cases with defects in the test case database to the total number of software defects is less than a second threshold, it is determined that the current state of the test case database satisfies the first condition.

6. The method according to claim 1, characterized in that The second mode is used to instruct to generate a test case based on the second corpus database; The generating a plurality of second test cases based on the second method comprises: Performing data enhancement on the second corpus database, and performing test case recommendation processing based on the second corpus database after the data enhancement to obtain multiple recommended test cases; Determine to obtain multiple generated test cases based on the updated code data, the requirement text, and the multiple recommended test cases; Obtain reference use cases that match updated business and updated requirements; A plurality of second test cases are generated according to the plurality of recommended test cases, the plurality of generated test cases and the reference case.

7. The method according to claim 6, characterized in that The step of generating a plurality of second test cases according to the plurality of recommended test cases, the plurality of generated test cases and the reference test cases comprises: Determine a fourth similarity between each of the recommended test cases and the corresponding test requirements, and use the plurality of recommended test cases with the highest fourth similarities as target recommended test cases; Determine a fifth similarity between each of the generated test cases and the corresponding test requirements, and use the multiple generated test cases with the highest fifth similarities as target generated test cases; Detecting each of the reference use cases to obtain the number of errors in each of the reference use cases, and taking a plurality of the reference use cases with the least number of errors as target reference use cases; Deduplication is performed on each of the target recommended test cases, the target generated test cases, and the target reference test cases; The plurality of second test cases are obtained by splicing together in the following manner: extracting at least one part from the target recommended test case, the target generated test case and the target reference test case after deduplication, and splicing the extracted parts into a complete second test case.

8. The method according to claim 7, characterized in that The method further comprises: Based on the multiple recommended test cases with the highest fourth similarity, the multiple generated test cases with the highest fifth similarity, and the multiple reference cases with the least number of errors, determine multiple third samples, and perform data enhancement on the multiple third samples to obtain multiple second positive samples; Based on the multiple recommended test cases with the lowest fourth similarity, the multiple generated test cases with the lowest fifth similarity, and the multiple reference cases with the largest number of errors, multiple fourth samples are determined, and noise is injected into the multiple fourth samples to obtain multiple second negative samples; The model to be trained is trained based on the labeled multiple second positive samples and the multiple second negative samples to obtain the large language model, and the large language model is used to perform case generation processing to generate test cases.

9. The method according to claim 1, characterized in that: The method further comprises: Determine a first accuracy rate, a first precision rate, and an evaluation value corresponding to the current test case database; When the first accuracy, the first precision and the evaluation value are all greater than a second ratio threshold, and the ratio of the second number to the total number of software defects is less than a third ratio threshold, it is determined that the current state of the test case database meets the second condition.

10. The method according to claim 1, characterized in that The method further comprises: Determine a first accuracy rate, a first precision rate, and an evaluation value corresponding to the current test case database; When at least one of the first accuracy, the first precision and the evaluation value is less than a fourth ratio threshold, or the ratio between the second number and the total number of software defects is greater than the first ratio threshold, it is determined that the current state of the test case database satisfies the third condition.

11. A test case generating device, characterized in that: The device comprises: A first test case generation module, configured to generate a plurality of first test cases based on a first method; and add the plurality of first test cases to a test case database; A second test case generation module is used to generate a plurality of second test cases based on a second method when the current state of the test case database satisfies the first condition; and store the plurality of second test cases in the test case database; A third use case generation module is used to generate test cases for the test case database based on a third method when the current state of the test case database meets the second condition; and to execute the step of generating multiple first test cases based on the first method when the current state of the test case database meets the third condition.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the test case generation method according to any one of claims 1 to 10 when executing the computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the test case generation method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the test case generation method according to any one of claims 1 to 10 is implemented.

Images