Test case generation method, device, equipment and readable storage medium

Abstract

A test case generation method, device, equipment and readable storage medium, the test case generation method comprising: using a large language model to generate a preset number of test cases for a test code; using a classification model to perform a quality assessment on each test case to obtain a quality score for each test case; if the number of qualified test cases is greater than or equal to a preset required number, then selecting a preset required number of test cases from the qualified test cases as recommended test cases for the test code; if the number of qualified test cases is less than the preset required number, then returning the unqualified test cases to the large language model for modification to obtain recommended test cases for the preset required number of test codes. Through this application, the current problem of low efficiency and easy omissions in the manual method of generating test cases is avoided, which not only gives full play to the advantage of the high efficiency of generating test cases by the large language model, but also ensures the quality of the test cases generated by the large language model.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to a test case generation method, device, equipment and readable storage medium. Background Art

[0002] A unit test case is a set of detailed instructions designed for the smallest test unit (usually a function or method) in software development, including input, execution conditions, test steps, and expected results. Unit testing plays a key role in the software development process. By accurately and meticulously testing each functional module, defects in the software can be discovered early and the reliability and quality of the code can be improved. There are many challenges in designing and implementing high-quality unit test cases, such as ensuring comprehensive test coverage of functions or methods, balancing the depth and breadth of testing, and simulating the behavior of dependent objects. These challenges require testers to have deep expertise and a deep understanding of the internal structure of the software being tested to ensure the effectiveness and efficiency of the test. Therefore, quickly and effectively generating high-quality unit test cases is of great value in improving the development speed and product quality of software projects.

[0003] Testers can manually design test scenarios based on requirement documents and design documents to generate unit test cases. Manual methods can accurately test specific requirements, but they are inefficient and prone to omissions. Large language models can quickly and efficiently generate a variety of unit test cases based on existing code bases and specifications, greatly improving the efficiency of unit test case generation. However, the quality of unit test cases generated by large language models varies greatly. If poor-quality unit test cases are recommended to testers, the trust of testers in test cases generated using large language models will be reduced, affecting the continued use and promotion of large language models by testers.

[0004] In summary, the current manual method of generating test cases is inefficient and prone to omissions, and the use of large language models makes it difficult to ensure the quality of the generated test cases. Summary of the invention

[0005] The present application provides a test case generation method, apparatus, device and readable storage medium, aiming to solve the technical problems that the current manual generation of test cases is inefficient and prone to omissions, and the quality of the generated test cases is difficult to ensure when using a large language model.

[0006] In a first aspect, an embodiment of the present application provides a test case generation method, the test case generation method comprising:

[0007] Use a large language model to generate a preset number of test cases for the test code;

[0008] Use the classification model to evaluate the quality of each test case and obtain the quality score of each test case;

[0009] Test cases with quality scores greater than or equal to the preset qualified scores are qualified test cases, and test cases with quality scores less than the preset qualified scores are unqualified test cases;

[0010] If the number of qualified test cases is greater than or equal to the preset required number, then selecting the preset required number of test cases from the qualified test cases as recommended test cases for the test code, and the preset required number is less than or equal to the preset generated number;

[0011] If the number of qualified test cases is less than the preset required number, the unqualified test cases are returned to the large language model for modification to obtain recommended test cases for the preset required number of test codes.

[0012] Optionally, returning the unqualified test cases to the large language model for modification to obtain recommended test cases of a preset required number of test codes includes:

[0013] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0014] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0015] If the number of qualified test cases is still less than the preset required number, returning to the step of returning the unqualified test cases to the large language model for modification to obtain modified test cases;

[0016] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0017] Optionally, returning the unqualified test cases to the large language model for modification to obtain recommended test cases of a preset required number of test codes also includes:

[0018] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0019] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0020] If the number of qualified test cases is still less than the preset required number and the number of modifications is less than or equal to the preset number of modifications, then return to the step of returning the unqualified test cases to the large language model for modification to obtain the modified test cases;

[0021] If the number of qualified test cases is still less than the preset required number and the number of modifications is greater than the preset number of modifications, then the preset required number of test cases are selected from all previously generated test cases in descending order of quality scores as recommended test cases for the test code;

[0022] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0023] Optionally, before using the classification model to perform quality assessment on each test case to obtain a quality score for each test case, the following steps are included:

[0024] Get multiple training and testing codes;

[0025] For each training test code, use the large language model to generate multiple training test cases for the training test code, and form a first training set with each training test code and the corresponding multiple training test cases;

[0026] For each training test case in the first training set, mark it as qualified or unqualified;

[0027] For each training test case in the first training set, extract code quality statistical features of the training test case, extract semantic features of the combination of the training test case and the training test code to which it belongs, and fuse the code quality statistical features and the semantic features to obtain fused features of the training test case;

[0028] The second training set is composed of the fused features and corresponding labels of each training test case;

[0029] The classification model is trained using the second training set. When the loss function of the classification model converges, the model at which the loss function converges is used as the trained classification model.

[0030] Optionally, the use of the classification model to perform quality assessment on each test case to obtain a quality score for each test case includes:

[0031] Extract code quality statistical features of each test case;

[0032] Extract semantic features of the combination of each test case and test code;

[0033] The code quality statistical features and semantic features of each test case are integrated to obtain the integrated features of each test case;

[0034] The fused features of each test case are input into the classification model for quality assessment to obtain the quality score of each test case.

[0035] Optionally, returning the unqualified test case to the large language model for modification includes:

[0036] The failed test cases and the code quality statistical features of each failed test case are returned to the large language model for modification.

[0037] Optionally, the classification model adopts CodeBERT, and the loss function of CodeBERT adopts Focal Loss. The formula of Focal Loss is: ,in, The probability of predicting that the test case is of qualified quality, is the class balance parameter, which is used to adjust the importance of the qualified or unqualified quality classes. It is a focusing parameter used to adjust the degree of attention the model pays to easy-to-classify samples and difficult-to-classify samples.

[0038] In a second aspect, an embodiment of the present application provides a test case generation device, the test case generation device comprising:

[0039] A generation module, for generating a preset number of test cases for the test code using the large language model;

[0040] An evaluation module is used to use a classification model to perform quality evaluation on each test case and obtain a quality score for each test case;

[0041] A determination module, used to determine test cases with quality scores greater than or equal to a preset qualified score as qualified test cases, and determine test cases with quality scores less than the preset qualified score as unqualified test cases;

[0042] A recommendation module, configured to select a preset number of test cases from the qualified test cases as recommended test cases for the test code if the number of qualified test cases is greater than or equal to a preset number of requirements, and the preset number of requirements is less than or equal to a preset number of generation;

[0043] The modification module is used to return the unqualified test cases to the large language model for modification if the number of qualified test cases is less than the preset required number, so as to obtain recommended test cases for the test code of the preset required number.

[0044] In a third aspect, an embodiment of the present application provides a test case generation device, which includes a processor, a memory, and a test case generation program stored in the memory and executable by the processor, wherein when the test case generation program is executed by the processor, the steps of the test case generation method described above are implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a test case generation program is stored, wherein when the test case generation program is executed by a processor, the steps of the test case generation method described above are implemented.

[0046] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0047] In an embodiment of the present application, a large language model is used to generate a preset number of test cases for a test code; a classification model is used to perform quality assessment on each test case to obtain a quality score for each test case; test cases with a quality score greater than or equal to a preset qualified score are qualified test cases, and test cases with a quality score less than the preset qualified score are unqualified test cases; if the number of qualified test cases is greater than or equal to a preset required number, a preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code, and the preset required number is less than or equal to a preset generated number; if the number of qualified test cases is less than the preset required number, the unqualified test cases are returned to the large language model for modification to obtain recommended test cases for the test code with a preset required number. The embodiment of the present application generates test cases using a large language model and uses a classification model to evaluate the quality of the test cases. If the number of qualified test cases meets a preset requirement, the preset number of test cases are selected from the qualified test cases as recommended test cases for the test code. If the number of qualified test cases does not meet the preset requirement, the unqualified test cases are returned to the large language model for modification, thereby obtaining recommended test cases for the preset number of test codes. The problem of low efficiency and easy omissions in the current manual method of generating test cases is avoided, and the advantage of high efficiency of generating test cases by the large language model is fully utilized, and the quality of the test cases generated by the large language model can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of an embodiment of a test case generation method of the present application;

[0049] Figure 2 A schematic diagram of a classification model training process in accordance with an embodiment of a test case generation method of the present application;

[0050] Figure 3 For this application Figure 1 Detailed flow chart of step S20;

[0051] Figure 4 This is a functional module diagram of an embodiment of a test case generation device of the present application;

[0052] Figure 5 A schematic diagram of the hardware structure of a test case generation device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] In a first aspect, an embodiment of the present application provides a test case generation method.

[0056] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a test case generation method of the present application. Figure 1 As shown, the test case generation method includes:

[0057] Step S10: Use the large language model to generate a preset number of test cases for the test code.

[0058] In this embodiment, large language models such as CodeLlama and ChatGPT can be used to generate a preset number of test cases for the test code, where the Prompt provided to the large language model is, for example: Please generate n non-repetitive test cases for the following test code y. Please ensure that the generated test cases meet the grammatical requirements, can run correctly, and have high code readability, maintainability, reusability and code coverage.

[0059] Step S20: Use the classification model to perform quality assessment on each test case to obtain a quality score for each test case.

[0060] In this embodiment, each generated test case is input into the classification model, the classification model performs quality assessment on each test case, and the classification model outputs the probability that each test case is a high-quality test case. The probability that each test case output by the classification model is a high-quality test case is regarded as the quality score of each test case.

[0061] Step S30, the test cases whose quality scores are greater than or equal to the preset qualified scores are qualified test cases, and the test cases whose quality scores are less than the preset qualified scores are unqualified test cases.

[0062] In this embodiment, for example, the preset qualified score can be set to 0.5. For each test case, when the quality score of the test case is greater than or equal to 0.5, it is judged as a qualified test case. Conversely, when the quality score of the test case is less than 0.5, it is judged as an unqualified test case.

[0063] Step S40, if the number of qualified test cases is greater than or equal to the preset required number, then select the preset required number of test cases from the qualified test cases as recommended test cases for the test code, and the preset required number is less than or equal to the preset generated number.

[0064] In this embodiment, taking the preset generation quantity of 10 and the preset requirement quantity of 8 as an example, that is, using the large language model to generate 10 test cases for the test code, the number of high-quality test cases required is 8. If the number of test cases that pass the quality assessment of the 10 test cases generated by the large language model is 9, which is greater than or equal to the preset requirement quantity of 8, then the number of qualified test cases is sufficient to meet the demand, then 8 test cases are selected from the 9 qualified test cases as recommended test cases for the test code. The specific selection method is not limited here. Preferably, the first 8 test cases can be selected from the 9 qualified test cases in order of quality scores from high to low as recommended test cases for the test code. The preset requirement quantity is less than or equal to the preset generation quantity, that is, the number of test cases generated by the large language model for the test code needs to be greater than or equal to the requirement quantity to ensure the number of test cases available for selection.

[0065] Step S50: If the number of qualified test cases is less than the preset required number, the unqualified test cases are returned to the large language model for modification to obtain recommended test cases for the preset required number of test codes.

[0066] In this embodiment, the large language model may not be able to generate 8 high-quality test cases at a time. If the number of qualified test cases among the 10 test cases generated by the large language model after quality assessment is 6, which is less than the preset required number of 8, the number of qualified test cases cannot meet the demand, and then the 4 unqualified test cases are returned to the large language model for modification. The prompt (prompt word) provided to the large language model is, for example: The 4 test cases you generated for the test code y are judged to be low-quality test cases. Please modify the test case. Please ensure that the generated test case meets the grammatical requirements, can run correctly, and has high code readability, maintainability, reusability and code coverage. The large language model will modify the 4 unqualified test cases and regenerate 4 test cases to continue to recommend a number of high-quality test cases that meet the requirements for the test code.

[0067] In this embodiment, large language models such as CodeLlama and ChatGPT can be used to generate a preset number of test cases for the test code, and each generated test case is input into the classification model. The classification model performs quality assessment on each test case, and the classification model outputs the probability that each test case is a high-quality test case. The probability that each test case output by the classification model is a high-quality test case is regarded as the quality score of each test case. The preset qualified score can be set to 0.5. For each test case, when the quality score of the test case is greater than or equal to 0.5, it is judged as a qualified test case. Conversely, when the quality score of the test case is less than 0.5, it is judged as an unqualified test case. If the number of test cases that are qualified after quality assessment of the preset number of test cases generated by the large language model is greater than or equal to the preset required number, the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code. The large language model may not be able to generate the preset required number of high-quality test cases at one time. If the number of test cases that are qualified after quality assessment of the preset number of test cases generated by the large language model is less than the preset required number, the unqualified test cases are returned to the large language model for modification. The large language model will modify the unqualified test cases and regenerate test cases to continue to recommend the required number of high-quality test cases for the test code. This avoids the problem that the current manual method of generating test cases is inefficient and prone to omissions, which not only gives full play to the high efficiency of the large language model in generating test cases, but also ensures the quality of the test cases generated by the large language model.

[0068] Further, in one embodiment, returning the unqualified test cases to the large language model for modification to obtain the recommended test cases of the preset required number of test codes includes:

[0069] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0070] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0071] If the number of qualified test cases is still less than the preset required number, returning to the step of returning the unqualified test cases to the large language model for modification to obtain modified test cases;

[0072] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0073] In this embodiment, taking the preset generation quantity of 10, the preset requirement quantity of 8, the number of qualified test cases of 6, and the number of unqualified test cases of 4 as an example, the 4 unqualified test cases are returned to the large language model for modification, and the large language model modifies the 4 unqualified test cases to regenerate 4 test cases, and then uses the classification model to perform quality assessment on the modified 4 test cases to obtain quality scores of the modified 4 test cases. Assuming that 1 of the 4 modified test cases becomes a qualified test case, the number of qualified test cases increases to 7 (plus the previous number of qualified test cases of 6). At this time, the number of qualified test cases is 7, which is still less than the preset requirement quantity of 8, then 3 unqualified test cases are returned to the large language model for further modification until the number of qualified test cases reaches 8. After the number of qualified test cases reaches 8, the 8 qualified test cases are used as recommended test cases for the test code.

[0074] Furthermore, in one embodiment, returning the unqualified test cases to the large language model for modification to obtain the recommended test cases of the preset required number of test codes also includes:

[0075] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0076] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0077] If the number of qualified test cases is still less than the preset required number and the number of modifications is less than or equal to the preset number of modifications, then return to the step of returning the unqualified test cases to the large language model for modification to obtain the modified test cases;

[0078] If the number of qualified test cases is still less than the preset required number and the number of modifications is greater than the preset number of modifications, then the preset required number of test cases are selected from all previously generated test cases in descending order of quality scores as recommended test cases for the test code;

[0079] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0080] In this embodiment, taking the preset generation quantity as 10, the preset requirement quantity as 8, the number of qualified test cases as 6, and the number of unqualified test cases as 4 as an example, the 4 unqualified test cases are returned to the large language model for modification, and the large language model will modify the 4 unqualified test cases to regenerate 4 test cases, and then use the classification model to perform quality assessment on the modified 4 test cases to obtain the quality scores of the modified 4 test cases. Assuming that 1 of the 4 modified test cases becomes a qualified test case, the number of qualified test cases increases to 7 (plus the previous number of qualified test cases of 6). At this time, the number of qualified test cases is 7, which is still less than the preset requirement quantity of 8. Since it takes time to modify test cases using a large language model, it is even possible that the quality of the test cases is still unqualified after multiple modifications. Therefore, in order to avoid this situation, , a judgment of the number of modifications is added here, and the preset number of modifications can be set according to specific needs. For example, the preset number of modifications is 2 times. If the number of qualified test cases 7 is still less than the preset number of requirements 8 and after 2 modifications, the number of modifications 2 is less than or equal to the preset number of modifications 2 times, then the unqualified test cases are returned to the large language model for further modification. If the number of qualified test cases 7 is still less than the preset number of requirements 8 and after 3 modifications, the number of modifications 3 is greater than the preset number of modifications 2 times, then the unqualified test cases are no longer returned to the large language model for modification. Instead, the preset number of test cases are selected from all the test cases generated in the past in order of quality scores from high to low as the recommended test cases for the test code. If the number of qualified test cases is greater than or equal to the preset number of requirements 8, then the preset number of test cases are selected from the qualified test cases as the recommended test cases for the test code.

[0081] Further, in one embodiment, referring to Figure 2 , Figure 2 This is a schematic diagram of the classification model training process of an embodiment of the test case generation method of the present application, such as Figure 2 As shown, before step S20, the following steps are included:

[0082] Step S01, obtaining multiple training and testing codes;

[0083] Step S02: for each training test code, use the large language model to generate multiple training test cases for the training test code, and form a first training set with each training test code and the corresponding multiple training test cases;

[0084] Step S03, marking each training test case in the first training set as qualified or unqualified;

[0085] Step S04, for each training test case in the first training set, extracting code quality statistical features of the training test case, extracting semantic features of the combination of the training test case and the training test code to which it belongs, and fusing the code quality statistical features and the semantic features to obtain fused features of the training test case;

[0086] Step S05, forming a second training set with the fusion features and corresponding labels of each training test case;

[0087] Step S06, using the second training set to train the classification model, when the loss function of the classification model converges, the model at the time of the loss function convergence is used as the trained classification model.

[0088] In this embodiment, in order to ensure the accuracy and efficiency of the classification model, the same large language model as in step S10 is used to prepare a training set for the classification model. Taking the large language model as an example, three training test codes are used to generate training test cases, and each training test code generates three training test cases. The obtained training test case data set S (i.e., the first training set) has a total of 9 training test cases, S = {t 11 , t 12 , t 13 , t 21 , t 22 , t 23 , t 31 , t 32 , t 33}; For each training test case in the first training set, domain experts can ij Perform quality assessment (the expert assessment criteria can mainly consider: whether the test case meets the syntax requirements, whether it can run correctly, the readability, maintainability, reusability and code coverage of the test case code, etc.), and mark the quality as qualified or unqualified. For example, qualified quality can be represented by "1" and unqualified quality can be represented by "0". The obtained labeled training test case data set can be represented as SF = { ( tij,F ) | tij∈S,F∈{0,1}}. Combined with the above example, SF = { (tij,F ) | tij∈S,F∈{0,1}} 11 ,0) , (t 12 ,1 ), (t 13 ,0),(t21 ,0) ,( t 22 ,0) , (t 23 ,0),(t 31 ,0) , (t 32 ,0) , (t 33,1)}, for each training test case code, use static analysis software such as SonarQube to extract code quality statistical features, the code quality statistical features include but are not limited to: test case code cyclomatic complexity, cognitive complexity, test case code repeated line ratio, test case code cohesion, the number of methods called by the test case code in the tested code, the number of repeated code lines between the test case code and the tested code, and the number of parameters passed between the test case code and the tested code. The training test code and training test cases can be run at the same time to calculate the training test case code line coverage, code branch coverage, and test case condition coverage.Among them, the cyclomatic complexity of the test code mainly reflects the number of paths in the code, that is, the number of branches. The higher the cyclomatic complexity, the more complex the logic in the code, which will increase the possibility of code errors and the difficulty of understanding and maintenance. The cognitive complexity mainly reflects the indicator of the difficulty of understanding the test code. The higher the complexity, the more complex the structure of the code, including more nested conditions and control flow elements, which will increase the difficulty of understanding and maintaining the code. The proportion of repeated lines of test code measures the proportion of repeated lines of code in the test code. More repeated code usually means lower maintainability. The cohesion of the test code is the degree of connection between the elements within the unit test code, that is, the degree of concentration of the code function. The higher the cohesion, the more concentrated and related the functions within the test code are, which is conducive to improving its functional efficiency and reliability. The number of methods called by the test code in the tested code measures the number of times the tested code method is called in the test code. This indicator reflects the degree of dependence of the test code on the tested code. The more method calls, the higher the coupling between the test and the tested code, which may increase the fragility and maintenance difficulty of the test code. The number of repeated lines of test code and tested code measures the number of repeated lines of code in the test code with the tested code. The number of lines of code, high number of repeated lines of code may indicate excessive coupling between the test code and the code under test, making it difficult to distinguish between the test logic and the code under test logic, increasing the maintenance complexity of the code base. The number of parameter passing between the test code and the code under test measures the number of parameters passed by the test code when calling the code under test. The more parameters passed, the more prior knowledge the test code needs to configure the code under test, reflecting a higher degree of coupling, which may not only reduce the readability of the test, but also affect the independence and flexibility of the test. The test code line coverage measures how many lines of the code under test are covered by the test code execution. Line coverage Coverage is the most basic form of coverage. The goal is to ensure that each line of code is executed at least once to detect potential execution errors. Test code branch coverage measures how many branches in the tested code are covered by the test code. Unlike line coverage, branch coverage focuses on the fact that both the true and false branches of each conditional judgment are executed at least once to ensure that all logical branches are verified. Test code condition coverage is similar to branch coverage, but more stringent, requiring that the possible results (true or false) of each conditional expression in each logical judgment be tested at least once. Condition coverage helps reveal errors that may be hidden in compound logical judgments. Among them, the classification model can use the CodeBERT model, for example. The CodeBERT model is an extension of the BERT model developed by Microsoft in 2020. It is a bimodal pre-training model for programming languages ​​(PL) and natural languages ​​(NL). It can perform downstream (NL-PL) tasks and use 6 programming languages ​​for NL-PL matching training. The code quality statistical features of the jth training test case corresponding to the training test code ci can be expressed as a vector F. tijThe CodeBERT model can be used to extract deep semantic features that reflect the quality of test cases from the training test code and training test cases. The CodeBERT model can automatically learn deep semantic features related to the quality of test cases from the source code dimension using a deep neural network, which complements the statistical features of code quality and more comprehensively evaluates the quality of test cases. In order to enable the CodeBERT model to better capture deep semantic features, corresponding tokens are added before the training test code and training test cases to indicate the content represented by the subsequent code text. The [tested code] token is added before the training test code, and the [test code] token is added before the training test case. The labeled test case dataset SF= { ( t ij ,F ) | t ij ∈S,F∈{0,1}}, we get the CodeBERT feature extraction dataset SFC = { ([tested code] ci [test code]tij ,Fc ) | tij∈S,F∈{0,1}}. SFC is input into the CodeBERT model for deep semantic feature extraction, and the test code ci corresponding to the jth test case t is obtained. ij The vector of deep semantic features is E tij Continuing with the above example, SFC=[tested code] SF={([test code]t 11 ,0), ([testcode]t 12 ,1), ([test code]t 13 ,0),([test code]t 21 ,0) ,([test code]t 22 ,0) ,([testcode]t 23 ,0),([test code]t 31 ,0) , ([test code]t 32 ,0) , ([test code]t 33 ,1)}. The code quality statistical features F of the training test cases tij Compared with the deep semantic features extracted by the CodeBERT model tij To avoid the code quality statistical features F of the training test cases, tij The deep semantic features E extracted by the CodeBERT model due to its small feature dimension tij Suppression, in the training test case code quality statistical features F tij Extracting deep semantic features with CodeBERT model tijBefore fusion, the fully connected layer is used to collect the code quality statistical features F of the training test cases tij Perform vector expansion to obtain the expanded statistical feature vector F' tij , so that its vector dimension is consistent with the deep semantic features E extracted by the CodeBERT model tij The expanded code quality statistical feature F' tij and deep semantic features E tij Fusion is performed to obtain the fusion feature V of the training test case tij , the fusion features and corresponding labels of each training test case form a second training set, which can be expressed as SV = { ( Vtij,F ) | F∈{0,1}}. The classification model is trained using the second training set. When the loss function of the classification model converges, the model at the time of convergence of the loss function is used as the trained classification model, or when the training times of the classification model reaches a certain preset times, the model at this time is used as the trained classification model.

[0089] Further, in one embodiment, referring to Figure 3 , Figure 3 For this application Figure 1 The detailed flow chart of step S20 is as follows: Figure 3 As shown, step S20 includes:

[0090] Step S201, extracting code quality statistical features of each test case;

[0091] Step S202, extracting semantic features of the combination of each test case and test code;

[0092] Step S203, fusing the code quality statistical features and semantic features of each test case to obtain the fusion features of each test case;

[0093] Step S204: Input the fusion features of each test case into the classification model for quality assessment to obtain a quality score for each test case.

[0094] In this embodiment, when using the classification model to perform quality assessment on each test case, the code quality statistical features of each test case can be first extracted through static analysis software such as SonarQube, and the semantic features of the combination of each test case and the test code can be extracted through the CodeBERT model. The code quality statistical features and semantic features of each test case are fused to obtain the fused features of each test case, and the fused features of each test case are input into the classification model for quality assessment to obtain the quality score of each test case.

[0095] Further, in one embodiment, returning the unqualified test case to the large language model for modification includes:

[0096] The failed test cases and the code quality statistical features of each failed test case are returned to the large language model for modification.

[0097] In this embodiment, in order to improve the quality of the test cases modified and regenerated by the large language model, when returning the unqualified test cases to the large language model for modification, the code quality statistical features of each unqualified test case are also returned to the large language model at the same time. At this time, the prompt provided to the large language model is, for example: the test case wi you generated for the test code y, the code cyclomatic complexity of wi is xi1, the cognitive complexity is xi2, the proportion of repeated lines of test case code is xi3, the cohesion of the test case code is xi4, the number of methods called by the test case code in the tested code is xi5, the number of repeated lines of code between the test case code and the tested code is xi6, the number of parameters passed between the test case code and the tested code is xi7, the test case code line coverage rate is xi8, the test case code branch coverage rate is xi9, and the test case condition coverage rate is xi10. This test case is judged as a low-quality test case. Please modify the test case and regenerate the test case to ensure that it meets the grammatical requirements, can run correctly, and has high code readability, maintainability, reusability and code coverage. The benefits of attaching code quality statistical features to large language models include: Targeted optimization: By providing large language models with information about which specific aspects make test cases judged as low quality, the model can be helped to make improvements more targeted. For example, if a test case is rated as low quality because of high code repetition, the model can focus on reducing duplicate code to optimize the test case; Enhanced model understanding: Providing these specific code and test quality features helps the model better understand the structure and quality standards of test cases, thereby generating high-quality test cases that better meet actual software testing needs; Improved generation quality: Accurate feature feedback can teach the model how to improve test cases in multiple dimensions. Improve its output, so as to improve not just one aspect but the overall quality of test cases. For example, by emphasizing code coverage, the test cases generated by the model can more effectively cover different branches and conditions of the tested code. Iterative learning: This feedback mechanism can be regarded as a teaching process, in which the model iteratively learns and improves by continuously receiving specific feedback on its output, and is ultimately able to independently generate high-quality test cases. Measurable improvement indicators: Using specific quality parameters as model input makes the improvement process quantifiable and traceable, and researchers and developers can more easily monitor the improvement of model performance and determine which parameters are most critical to improving test case quality.

[0098] Furthermore, in one embodiment, the classification model adopts CodeBERT, the loss function of CodeBERT adopts Focal Loss, and the formula of Focal Loss is: ,in, The probability of predicting that the test case is of qualified quality, is the class balance parameter, which is used to adjust the importance of the qualified or unqualified quality classes. It is a focusing parameter used to adjust the degree of attention the model pays to easy-to-classify samples and difficult-to-classify samples.

[0099] In this embodiment, It is a category balance parameter used to adjust the importance of qualified or unqualified quality categories. Especially in the case of category imbalance, it is generally set to a value greater than 0 and less than 1. For example, it can be set to 0.25 in this embodiment. It is a focusing parameter used to adjust the model's attention to easy-to-classify samples and difficult-to-classify samples. The larger its value, the lower the attention to easy-to-classify samples. For example, it can be set to 0.99 in this embodiment. Since large language models generally generate fewer high-quality test cases, high-quality test cases in the labeled test case dataset are samples that are difficult for the CodeBERT model to classify, and low-quality test cases in the labeled test case dataset are samples that are easy to classify for the CodeBERT model. Using the Focal Loss function helps the test case quality assessment model focus more on high-quality test cases.

[0100] In a second aspect, an embodiment of the present application also provides a test case generating device.

[0101] In one embodiment, referring to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of a test case generation device of the present application, such as Figure 4 As shown, the test case generating device includes:

[0102] A generating module 10, configured to generate a preset number of test cases for the test code using the large language model;

[0103] An evaluation module 20 is used to perform quality evaluation on each test case using a classification model to obtain a quality score for each test case;

[0104] A determination module 30, configured to determine test cases with a quality score greater than or equal to a preset qualified score as qualified test cases, and determine test cases with a quality score less than the preset qualified score as unqualified test cases;

[0105] A recommendation module 40, configured to select a preset number of test cases from the qualified test cases as recommended test cases for the test code if the number of qualified test cases is greater than or equal to a preset number of requirements, and the preset number of requirements is less than or equal to a preset number of generation;

[0106] The modification module 50 is used to return the unqualified test cases to the large language model for modification if the number of qualified test cases is less than the preset required number, so as to obtain recommended test cases for the test code of the preset required number.

[0107] Furthermore, in one embodiment, the modification module 50 is further configured to:

[0108] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0109] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0110] If the number of qualified test cases is still less than the preset required number, returning to the step of returning the unqualified test cases to the large language model for modification to obtain modified test cases;

[0111] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0112] Furthermore, in one embodiment, the modification module 50 is further configured to:

[0113] Return unqualified test cases to the large language model for modification to obtain modified test cases;

[0114] Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case;

[0115] If the number of qualified test cases is still less than the preset required number and the number of modifications is less than or equal to the preset number of modifications, then return to the step of returning the unqualified test cases to the large language model for modification to obtain the modified test cases;

[0116] If the number of qualified test cases is still less than the preset required number and the number of modifications is greater than the preset number of modifications, then the preset required number of test cases are selected from all previously generated test cases in descending order of quality scores as recommended test cases for the test code;

[0117] If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

[0118] Furthermore, in one embodiment, the test case generating device further includes a training module, which is used to:

[0119] Get multiple training and testing codes;

[0120] For each training test code, use the large language model to generate multiple training test cases for the training test code, and form a first training set with each training test code and the corresponding multiple training test cases;

[0121] For each training test case in the first training set, mark it as qualified or unqualified;

[0122] For each training test case in the first training set, extract code quality statistical features of the training test case, extract semantic features of the combination of the training test case and the training test code to which it belongs, and fuse the code quality statistical features and the semantic features to obtain fused features of the training test case;

[0123] The second training set is composed of the fused features and corresponding labels of each training test case;

[0124] The classification model is trained using the second training set. When the loss function of the classification model converges, the model at which the loss function converges is used as the trained classification model.

[0125] Furthermore, in one embodiment, the evaluation module 20 is further configured to:

[0126] Extract code quality statistical features of each test case;

[0127] Extract semantic features of the combination of each test case and test code;

[0128] The code quality statistical features and semantic features of each test case are integrated to obtain the integrated features of each test case;

[0129] The fused features of each test case are input into the classification model for quality assessment to obtain the quality score of each test case.

[0130] Furthermore, in one embodiment, the modification module 50 is further configured to:

[0131] The failed test cases and the code quality statistical features of each failed test case are returned to the large language model for modification.

[0132] Furthermore, in one embodiment, the classification model adopts CodeBERT, the loss function of CodeBERT adopts Focal Loss, and the formula of Focal Loss is: ,in, The probability of predicting that the test case is of qualified quality, is the class balance parameter, which is used to adjust the importance of the qualified or unqualified quality classes. It is a focusing parameter used to adjust the degree of attention the model pays to easy-to-classify samples and difficult-to-classify samples.

[0133] Among them, the functional implementation of each module in the above-mentioned test case generation device corresponds to the various steps in the above-mentioned test case generation method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0134] In a third aspect, an embodiment of the present application provides a test case generating device.

[0135] Reference Figure 5 , Figure 5 The hardware structure diagram of the test case generation device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the test case generation device may include a processor, a memory, a communication interface, and a communication bus.

[0136] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0137] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces, which are used to interconnect devices within the test case generation device, as well as interfaces used to interconnect the test case generation device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, optical fiber interfaces, ATM interfaces, etc.; user devices can be display screens (Display), keyboards (Keyboard), etc.

[0138] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0139] The processor may be a general-purpose processor, and the general-purpose processor may call the test case generation program stored in the memory and execute the test case generation method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the test case generation program is called may refer to the various embodiments of the test case generation method of the present application, and will not be repeated here.

[0140] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0141] In a fourth aspect, an embodiment of the present application also provides a readable storage medium.

[0142] A test case generation program is stored on the readable storage medium of the present application, wherein when the test case generation program is executed by a processor, the steps of the test case generation method as described above are implemented.

[0143] Among them, the method implemented when the test case generation program is executed can refer to the various embodiments of the test case generation method of the present application, and will not be repeated here.

[0144] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0145] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0146] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0147] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0148] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0150] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A test case generation method, characterized in that: The test case generation method comprises: Use a large language model to generate a preset number of test cases for the test code; Use the classification model to evaluate the quality of each test case and obtain the quality score of each test case; Test cases with quality scores greater than or equal to the preset qualified scores are qualified test cases, and test cases with quality scores less than the preset qualified scores are unqualified test cases; If the number of qualified test cases is greater than or equal to the preset required number, then selecting the preset required number of test cases from the qualified test cases as recommended test cases for the test code, and the preset required number is less than or equal to the preset generated number; If the number of qualified test cases is less than the preset required number, the unqualified test cases are returned to the large language model for modification to obtain recommended test cases for the preset required number of test codes; Before using the classification model to perform quality assessment on each test case and obtain the quality score of each test case, the following steps are included: Get multiple training and testing codes; For each training test code, use the large language model to generate multiple training test cases for the training test code, and form a first training set with each training test code and the corresponding multiple training test cases; For each training test case in the first training set, mark it as qualified or unqualified; For each training test case in the first training set, extract code quality statistical features of the training test case, extract semantic features of the combination of the training test case and the training test code to which it belongs, and fuse the code quality statistical features and the semantic features to obtain fused features of the training test case; The second training set is composed of the fused features and corresponding labels of each training test case; The classification model is trained using the second training set. When the loss function of the classification model converges, the model at which the loss function converges is used as the trained classification model.

2. The test case generation method according to claim 1, characterized in that: The method of returning the unqualified test cases to the large language model for modification to obtain the recommended test cases of the preset required number of test codes includes: Return unqualified test cases to the large language model for modification to obtain modified test cases; Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case; If the number of qualified test cases is still less than the preset required number, returning to the step of returning the unqualified test cases to the large language model for modification to obtain modified test cases; If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

3. The test case generation method according to claim 1, characterized in that: The step of returning the unqualified test cases to the large language model for modification to obtain the recommended test cases of the preset required number of test codes also includes: Return unqualified test cases to the large language model for modification to obtain modified test cases; Use the classification model to evaluate the quality of each modified test case to obtain a quality score for each modified test case; If the number of qualified test cases is still less than the preset required number and the number of modifications is less than or equal to the preset number of modifications, then return to the step of returning the unqualified test cases to the large language model for modification to obtain the modified test cases; If the number of qualified test cases is still less than the preset required number and the number of modifications is greater than the preset number of modifications, then the preset required number of test cases are selected from all previously generated test cases in descending order of quality scores as recommended test cases for the test code; If the number of qualified test cases is greater than or equal to the preset required number, then the preset required number of test cases are selected from the qualified test cases as recommended test cases for the test code.

4. The test case generation method according to claim 1, characterized in that: The quality assessment of each test case using the classification model to obtain the quality score of each test case includes: Extract code quality statistical features of each test case; Extract semantic features of the combination of each test case and test code; The code quality statistical features and semantic features of each test case are integrated to obtain the integrated features of each test case; The fused features of each test case are input into the classification model for quality assessment to obtain the quality score of each test case.

5. The test case generation method according to claim 4, characterized in that: The returning the unqualified test cases to the large language model for modification includes: The failed test cases and the code quality statistical features of each failed test case are returned to the large language model for modification.

6. The test case generation method according to claim 1, characterized in that: The classification model adopts CodeBERT, and the loss function of CodeBERT adopts Focal Loss. The formula of Focal Loss is: ,in, The probability of predicting that the test case is of qualified quality, is the class balance parameter, which is used to adjust the importance of the qualified or unqualified quality classes. It is a focusing parameter used to adjust the model's attention to easy-to-classify samples and difficult-to-classify samples.

7. A test case generating device, characterized in that: The test case generating device comprises: A generation module, for generating a preset number of test cases for the test code using the large language model; An evaluation module is used to use a classification model to perform quality evaluation on each test case and obtain a quality score for each test case; A determination module, used to determine test cases with quality scores greater than or equal to a preset qualified score as qualified test cases, and determine test cases with quality scores less than the preset qualified score as unqualified test cases; A recommendation module, configured to select a preset number of test cases from the qualified test cases as recommended test cases for the test code if the number of qualified test cases is greater than or equal to a preset number of requirements, and the preset number of requirements is less than or equal to a preset number of generation; A modification module, for returning unqualified test cases to the large language model for modification if the number of qualified test cases is less than the preset required number, so as to obtain recommended test cases for the test code of the preset required number; The test case generating device also includes a training module, which is used to: Get multiple training and testing codes; For each training test code, use the large language model to generate multiple training test cases for the training test code, and form a first training set with each training test code and the corresponding multiple training test cases; For each training test case in the first training set, mark it as qualified or unqualified; For each training test case in the first training set, extract code quality statistical features of the training test case, extract semantic features of the combination of the training test case and the training test code to which it belongs, and fuse the code quality statistical features and the semantic features to obtain fused features of the training test case; The second training set is composed of the fused features and corresponding labels of each training test case; The classification model is trained using the second training set. When the loss function of the classification model converges, the model at which the loss function converges is used as the trained classification model.

8. A test case generation device, characterized in that: The test case generation device includes a processor, a memory, and a test case generation program stored in the memory and executable by the processor, wherein when the test case generation program is executed by the processor, the steps of the test case generation method as described in any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a test case generation program, wherein when the test case generation program is executed by a processor, the steps of the test case generation method according to any one of claims 1 to 6 are implemented.

Images