Test case generation method and system

Abstract

The invention discloses a test case generation method and device and a medium, and the generation method comprises the steps: obtaining a first code context, an intention instruction corresponding to the first code context, a first prompt word and a language model; inputting the first code context, the intention instruction and the first prompt into a language model to obtain a first code intention and a first test case; based on different sampling temperatures, inputting the first code context, the first code intention and the first prompt into a language model to obtain a plurality of second test cases; obtaining a third test case according to the first test case and all the second test cases; and inputting the third test case and the second prompt into a language model to obtain a target test case. According to the generation method, the computing resources and time required for generating the test case can be reduced, and the readability, quality and test coverage of the test case are improved. The invention relates to the technical field of deep learning.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a test case generation method and system. Background Art

[0002] Unit testing is a basic component of the software development life cycle. It mainly checks and verifies the testable units in the software through generated test cases. The testable unit can be the method to be tested (also known as the focus method), etc. It plays an important role in early software defect identification, reducing development costs, and improving software maintainability and scalability.

[0003] In the related art, unit testing methods can be roughly divided into two categories. The first type is usually based on unit test generation tools, such as Evosuite, which uses evolutionary algorithms to generate test cases and guides the search process through fitness functions. The test cases generated by this method have poor readability.

[0004] The second method is usually based on deep learning technology. Specifically, it pre-trains or fine-tunes a large language model to directly convert the given focus method source code into the corresponding test case. This method requires more computing resources and time, and is easily affected by training data. The quality of the generated test cases is poor and the test coverage is unsatisfactory.

[0005] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0006] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0007] To this end, an object of an embodiment of the present invention is to provide a test case generation method and system, wherein the generation method can reduce the computing resources and time required for test case generation, and improve the readability, quality and test coverage of the test cases.

[0008] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0009] In a first aspect, an embodiment of the present application provides a test case generation method, comprising:

[0010] Acquire a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to instruct generation of a code intention corresponding to the first code context, and the first prompt is used to instruct generation of a test case corresponding to the first code context;

[0011] Inputting the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case;

[0012] Based on different sampling temperatures, the first code context, the first code intent, and the first prompt are input into the language model to obtain a plurality of second test cases;

[0013] According to the first test case and all the second test cases, a third test case is obtained, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases;

[0014] The third test case and the second prompt are input into the language model to obtain a target test case, wherein the second prompt is used to instruct generation of a test code corresponding to the third test case.

[0015] In addition, the generation method according to the above embodiment of the present application may also have the following additional technical features:

[0016] Further, in one embodiment of the present application, the step of inputting the first code context, the intention instruction, and the first prompt into the language model to obtain a first code intention and a first test case corresponding to the first code context includes:

[0017] Inputting the first code context and the intention instruction into the language model to obtain the first code intention, where the first code intention is used to indicate the code intention for generating the first code context;

[0018] The first code context, the first code intent, and the first prompt are input into the language model to obtain the first test case.

[0019] Further, in one embodiment of the present application, based on different sampling temperatures, the first code context, the first code intent, and the first prompt are input into the language model to obtain multiple second test cases, including:

[0020] Constructing an initial prompt according to the first code context, the first code intention and the first prompt;

[0021] Based on different sampling temperatures, the initial prompt is sampled and updated by the language model to obtain a candidate prompt sequence, wherein the candidate prompt sequence includes a plurality of candidate prompts, each candidate prompt includes a first code context, a second code intent, and a third prompt word;

[0022] The first code context, the corresponding second code intention and the corresponding third prompt are input into the language model to obtain a plurality of the second test cases.

[0023] Further, in one embodiment of the present application, obtaining a third test case according to the first test case and all the second test cases includes:

[0024] Obtaining compilation and running results according to the first test case and all the second test cases;

[0025] According to the compilation and running result, performing a first screening on the first test case and all the second test cases to obtain a number of successfully compiled test cases;

[0026] The third test case is determined according to the sampling temperature corresponding to each of the successfully compiled test cases.

[0027] Further, in one embodiment of the present application, the step of inputting the third test case and the second prompt into the language model to obtain a target test case includes:

[0028] According to the second prompt, acquiring a semantic strategy and a variation strategy;

[0029] Inputting the semantic strategy and the third test case into the language model to obtain a first test input sequence;

[0030] According to the mutation strategy, the first test input sequence is mutated to obtain the target test case.

[0031] Furthermore, in the embodiment of the present application, the generating method further includes:

[0032] Get the preset time threshold;

[0033] Get the current loop time and the second test input sequence;

[0034] According to the mutation strategy, randomly mutate the second test input sequence to obtain a third test input sequence, wherein the number of test inputs in the second test input sequence of the current cycle round is greater than the number of test inputs in the second test input sequence of the previous cycle round;

[0035] If the current cycle time is greater than or equal to the time threshold, the target test case is obtained according to the third test input sequence; or, if the current cycle time is less than the time threshold, the current cycle time is updated, and the second test input sequence is updated according to the third test input sequence, and then the step of obtaining the current cycle time and the second test input sequence is returned.

[0036] Further, in the embodiment of the present application, obtaining a target test case according to the third test input sequence includes:

[0037] Determining a variation test code according to the third test input sequence;

[0038] Compile and verify the mutation test code to obtain a compilation verification result;

[0039] If the compilation verification result is that the variation test code is compiled successfully, coverage verification is performed on the variation test code according to a preset coverage threshold to obtain a coverage verification result;

[0040] If the coverage verification result is that the test coverage of the variant test code is greater than or equal to the coverage threshold, the variant test code is determined as the target test case.

[0041] Furthermore, in the embodiment of the present application, the generating method further includes:

[0042] Acquire original test data, where the original test data is the first test case, the second test case, or the variant test code;

[0043] The original test data is compiled and repaired using the language model to obtain target test data.

[0044] Furthermore, in the embodiment of the present application, the target test data is obtained by the following steps:

[0045] Get the invalid optimization threshold;

[0046] Acquire the current invalid optimization count and first test data, as well as compilation error information corresponding to the first test data and a second code context, where the second code context is used to represent a code context related to the compilation error information;

[0047] Constructing a repair prompt according to the compilation error information and the second code context, wherein the repair prompts corresponding to different loop rounds are different;

[0048] Inputting the first test data and the repair prompt into the language model to obtain second test data;

[0049] According to the first test data, optimizing and verifying the second test data to obtain an optimization and verification result;

[0050] If the optimization verification result is that there is no compilation error information for the second test data, the second test data is determined as the target test data; or, if the optimization verification result is that the number of compilation error information of the second test data is less than the number of compilation error information of the first test data, the first test data is updated according to the second test data, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned; or, if the optimization verification result is that the number of compilation error information of the second test data is greater than or equal to the number of compilation error information of the first test data and the current number of invalid optimizations is less than the invalid optimization threshold, the current number of invalid optimizations is updated, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned.

[0051] In a second aspect, an embodiment of the present application provides a test case generation system, including:

[0052] an acquisition unit, configured to acquire a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to instruct generation of a code intention corresponding to the first code context, and the first prompt is used to instruct generation of a test case corresponding to the first code context;

[0053] A first processing unit, configured to input the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case;

[0054] A second processing unit is configured to input the first code context, the first code intent, and the first prompt into the language model based on different sampling temperatures to obtain a plurality of second test cases;

[0055] A third processing unit, configured to obtain a third test case according to the first test case and all the second test cases, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases;

[0056] The fourth processing unit is used to input the third test case and the second prompt into the language model to obtain a target test case, wherein the second prompt is used to instruct the generation of a test code corresponding to the third test case.

[0057] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0058] at least one processor;

[0059] at least one memory for storing at least one program;

[0060] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0061] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0062] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0063] A test case generation method, device and medium disclosed in an embodiment of the present application, wherein the generation method obtains a first code context, an intention instruction corresponding to the first code context, a first prompt and a language model, wherein the intention instruction is used to indicate the generation of a code intention corresponding to the first code context, and the first prompt is used to indicate the generation of a test case corresponding to the first code context; the first code context, the intention instruction and the first prompt are input into the language model to obtain a first code intention and a first test case; based on different sampling temperatures, the first code context, the first code intention and the first prompt are input into the language model to obtain multiple second test cases; a third test case is obtained based on the first test case and all the second test cases, and the third test case is used to characterize the test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases; the third test case and the second prompt are input into the language model to obtain a target test case, and the second prompt is used to indicate the generation of a test code corresponding to the third test case. The generation method is based on intention instructions, which can enable the language model to understand the code intention of the focus method according to the first code context, and input the understood code intention as an intention prompt into the language model to generate test cases, which can generate test cases with better quality and readability; in addition, the generation method is based on multi-temperature sampling of the language model, selects the test case that is successfully compiled and run with the smallest sampling temperature (i.e., the third test case) among all test cases, and inputs the third test case as a few-sample prompt into the language model to generate the target test case, which can effectively improve the test coverage of the test case generation; in addition, the generation method generates the target test case corresponding to the code context based on the code intention and prompt, which does not require pre-training or fine-tuning of the large language model, and can effectively reduce the computing resources and time required for test case generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 A flowchart of a test case generation method provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of the structure of a test case generation system provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] In the related art, unit testing methods can be roughly divided into two categories. The first type is usually based on unit test generation tools, such as Evosuite, which uses evolutionary algorithms to generate test cases and guides the search process through fitness functions. The test cases generated by this method have poor readability.

[0071] The second method is usually based on deep learning technology, which is implemented by pre-training or fine-tuning the large language model. The fine-tuning is specifically to update the weight parameters of the large language model by further training on the data of a specific task. However, suitable data sets are difficult to find, and as the scale of the large language model continues to grow, the computing resources and time required to fine-tune the large language model are high. In addition, since the quality of test cases generated by the large language model is easily affected by the training data, directly converting the given focus method source code into the corresponding test case, the large language model may not be able to accurately understand the intent of the code when faced with overly complex, unclearly structured code, and code that has never been seen or is significantly different from the training data, thereby generating inaccurate or invalid test cases. The test coverage of the inaccurate or invalid test cases is often unsatisfactory, and the quality of the generated test cases is poor.

[0072] In view of this, an embodiment of the present invention provides a test case generation method. The generation method is based on intention instructions, which can enable a language model to understand the code intention of a focus method according to a first code context, and input the understood code intention as an intention prompt into the language model to generate a test case, which can generate a test case with better quality and readability; in addition, the generation method is based on multi-temperature sampling of the language model, selects the test case (i.e., the third test case) that is successfully compiled and run and has the lowest sampling temperature among all test cases, and inputs the third test case as a few-sample prompt into the language model to generate a target test case, which can effectively improve the test coverage of the test case generation; in addition, the generation method generates a target test case corresponding to the code context based on the code intention and the prompt, which does not require pre-training or fine-tuning of a large language model, and can effectively reduce the computing resources and time required for test case generation.

[0073] Reference Figure 1 In an embodiment of the present application, a test case generation method includes:

[0074] Step 110: Obtain a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to instruct the generation of a code intention corresponding to the first code context, and the first prompt is used to instruct the generation of a test case corresponding to the first code context;

[0075] In an embodiment of the present application, the language model may be a large language model (i.e., an LLM model), which may be a ChatGPT series model; the first code context may be a code context required for test case generation extracted from the source code of the method under test, and the first code context (Method Context) generally includes import statements (import / package), class declarations (Class declaration), constructor signatures (Constructors Signature), related fields (Fields), and focal methods (Focal Method), etc. The intention instruction is used to instruct the language model to generate the code intention of the first code context, which may require the language model to return a natural language description of the expected function of the method under test; the first prompt is used to instruct the language model to generate a corresponding test case for the focal method.

[0076] Step 120: input the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case;

[0077] In some embodiments, the step 120 of inputting the first code context, the intention instruction, and the first prompt into the language model to obtain a first code intention and a first test case corresponding to the first code context includes:

[0078] A1. Inputting the first code context and the intention instruction into the language model to obtain the first code intention, where the first code intention is used to indicate the code intention for generating the first code context;

[0079] A2. Input the first code context, the first code intent, and the first prompt into the language model to obtain the first test case.

[0080] In an embodiment of the present application, step A1 may be inputting the first code context and the intention instruction into the language model. Specifically, the intention instruction may be "You are a professional Java engineer. Based on the following code, please use one sentence to describe the function and intention of {focus method name} as concisely and comprehensively as possible", so that the language model outputs the intention of the focus method, that is, a natural language description of the expected function of the focus method, which can be determined as the first code intention corresponding to the first code context.

[0081] It can be understood that step A2 can be to use the first code intent as the intention prompt of the first code context, and then guide the language model to generate an initial test case (i.e., the first test case) for the focus method through the first prompt. The first prompt can specifically be "You are a professional Java test engineer. Please generate a test case that can be compiled and executed based on the above focus method code and its code intent."

[0082] Step 130: Based on different sampling temperatures, the first code context, the first code intent, and the first prompt are input into the language model to obtain a plurality of second test cases;

[0083] In some embodiments, the step 130, based on different sampling temperatures, inputting the first code context, the first code intent, and the first prompt into the language model to obtain a plurality of second test cases, includes:

[0084] B1. constructing an initial prompt according to the first code context, the first code intention and the first prompt;

[0085] B2. Based on different sampling temperatures, the initial prompt is sampled and updated by the language model to obtain a candidate prompt sequence, wherein the candidate prompt sequence includes a plurality of candidate prompts, each candidate prompt includes a first code context, a second code intent, and a third prompt;

[0086] B3. Input the first code context, the corresponding second code intention and the corresponding third prompt into the language model to obtain a plurality of the second test cases.

[0087] In an embodiment of the present application, after obtaining the first test case or the first code intent, multi-temperature sampling can be performed on the input initial prompt through a language model based on a prompt distillation (PD) to generate multiple candidate prompts and a test case corresponding to each candidate prompt, and each candidate prompt corresponds to a different sampling temperature.

[0088] It can be understood that step B1 can be to determine the first code context, the first code intent and the first prompt as the initial prompt, which may contain redundant or irrelevant information in practice; step B2 can be to use different sampling temperature settings, and the specific sampling temperature can be at least one of 0.3, 0.6, 0.8, 1, etc., to guide the language model to regenerate multiple more diverse and more complete candidate prompts with the initial prompt as a template to eliminate the impact of low-quality prompts on test case generation, specifically keep the first code context in the initial prompt unchanged, use the first code intent and the first prompt in the initial prompt as a few sample prompts to guide the language model to generate candidate prompts, and finally integrate all the obtained candidate prompts to obtain a candidate prompt sequence. Step B3 can be to extract candidate prompts from the candidate prompt sequence cyclically after obtaining the candidate prompt sequence, and input them into the language model, so as to generate a corresponding second test case for each candidate prompt.

[0089] Step 140: Obtain a third test case according to the first test case and all the second test cases, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases;

[0090] In some embodiments, the step 140 of obtaining a third test case according to the first test case and all the second test cases includes:

[0091] C1. Obtaining compilation and running results according to the first test case and all the second test cases;

[0092] C2. According to the compilation and running result, the first test case and all the second test cases are first screened to obtain a number of successfully compiled test cases;

[0093] C3. Determine the third test case according to the sampling temperature corresponding to each of the successfully compiled test cases.

[0094] In an embodiment of the present application, step C1 may be to compile and run the first test case and all the second test cases in a program running environment (such as a pycharm running environment) to obtain a compilation and running result corresponding to each test case, and the type of the compilation and running result may be two types: compilation success and compilation failure; step C2 may be to filter all the compilation and running results to obtain a compilation and running result of the type of successful compilation, and then, among the first test case and all the second test cases, determine the test case corresponding to the compilation and running result of the type of successful compilation as a successful compilation case.

[0095] It is understandable that the lower the sampling temperature of the language model, the more correct the test cases the language model tends to give, that is, the higher the confidence of the test cases given, the lower the diversity; and the higher the sampling temperature of the language model, the higher the diversity of the test cases given by the language model, and the lower the confidence. Therefore, in the embodiment of the present application, step C3 can be to determine the compiled successfully case with the lowest sampling temperature as the third test case among all the compiled successfully cases, and the third test case can improve the test coverage of the test case generation.

[0096] Step 150: Input the third test case and the second prompt into the language model to obtain a target test case, wherein the second prompt is used to instruct generation of a test code corresponding to the third test case.

[0097] In an embodiment of the present application, the second prompt is used to instruct the generation of test code corresponding to the third test case, and is specifically used to instruct the generation of test code with a larger test coverage than that corresponding to the third test case. Specifically, the third test case in step 150 is a small sample prompt input to the language model, and the second prompt can be "You are a professional Java test engineer. Please generate more test cases for {focus method name} based on the above content to improve test coverage, and output complete test code. There is no need to output any other irrelevant information."

[0098] In some embodiments, the step 150 of inputting the third test case and the second prompt into the language model to obtain a target test case includes:

[0099] D1. Acquire semantic strategy and variation strategy according to the second prompt;

[0100] D2. Inputting the semantic strategy and the third test case into the language model to obtain a first test input sequence;

[0101] In the embodiment of the present application, the semantic strategy is used to instruct the language model to generate a test input that is semantically similar to but not completely identical to the third test case; the mutation strategy is used to instruct the language model to mutate the test input in the test input sequence. Step D2 can be to input the third test case into the language model, guide the language model to generate several test inputs that are semantically similar to but not completely identical to the first test input sequence through the semantic strategy, so as to avoid the situation where the language model generates the same output after receiving the same input, and then construct the first test input sequence from the obtained several test inputs that are semantically similar but not completely identical.

[0102] D3. According to the mutation strategy, perform input mutation on the first test input sequence to obtain the target test case.

[0103] Furthermore, the generation method further comprises:

[0104] D31, obtaining a preset time threshold;

[0105] D32, get the current cycle time and the second test input sequence;

[0106] D33. According to the mutation strategy, randomly mutate the second test input sequence to obtain a third test input sequence, wherein the number of test inputs in the second test input sequence of the current cycle round is greater than the number of test inputs in the second test input sequence of the previous cycle round;

[0107] D34. If the current cycle time is greater than or equal to the time threshold, the target test case is obtained according to the third test input sequence; or, if the current cycle time is less than the time threshold, the current cycle time is updated, and the second test input sequence is updated according to the third test input sequence, and then the step of obtaining the current cycle time and the second test input sequence is returned.

[0108] In an embodiment of the present application, the time threshold is the maximum running time of the loop operation, and the specific time can be set according to actual conditions, for example, any one of 15 seconds, 30 seconds, 60 seconds, etc.; if the current loop round is equal to 1, the second test input sequence can be the first test input sequence; or, if the current loop round is greater than or equal to 2, the second test input sequence can be the third test input sequence of the previous loop round.

[0109] It can be understood that step D33 can be to randomly sample the test inputs in the second test input sequence, sample to obtain an original test input, and then mutate the original test input based on a mutation strategy. The specific mutation strategy can be maximum critical value mutation, minimum critical value mutation, abnormal input, blank input, special character input, etc., so as to obtain the target test input, and then put the generated target test input back into the original second test input sequence to obtain a third test input sequence, so that the number of test inputs in the second test input sequence of the current cycle round is greater than the number of test inputs in the second test input sequence of the previous cycle round, which can diversify the output of the language model.

[0110] It should be noted that if the current cycle time is greater than or equal to the time threshold, the loop can be exited, and the target test case can be determined based on the test input in the third test input sequence under the current cycle round; or, if the current cycle time is less than the time threshold, the current cycle time is updated according to the actual duration of the cycle, and the third test input sequence of the current cycle round is determined as the second test input sequence of the next cycle round, and then the process returns to execute step D32.

[0111] It is worth mentioning that the time threshold in the embodiment of the present application can also be simply replaced by a cycle number threshold, and the current cycle time can be correspondingly replaced by the current cycle round. The example of this application is for illustration only and does not limit the present application.

[0112] Furthermore, the step D34, obtaining a target test case according to the third test input sequence, includes:

[0113] D341. Determine a variation test code according to the third test input sequence;

[0114] D342, compile and verify the variation test code to obtain a compilation verification result;

[0115] D343. If the compilation verification result is that the variation test code is compiled successfully, coverage verification is performed on the variation test code according to a preset coverage threshold to obtain a coverage verification result;

[0116] D344. If the coverage verification result is that the test coverage of the variant test code is greater than or equal to the coverage threshold, the variant test code is determined as the target test case.

[0117] In an embodiment of the present application, step D341 can be to determine a number of variant test codes based on the code fragment corresponding to each test input in the third test input sequence, and each test input corresponds to a variant test code; the mutation verification in step D342 can be to verify whether all variant test codes can be compiled and run in the program running environment, so as to obtain a compilation verification result corresponding to each variant test code; the specific value of the coverage threshold can be set according to actual conditions, for example, any one of 0.6, 0.8, 0.9, 0.95, 0.98, etc., and step D343 can be based on coverage calculation tools such as Jacoco to perform test coverage calculation on the successfully compiled variant test codes, so as to obtain coverage verification results corresponding to each successfully compiled variant test code.

[0118] It can be understood that if all the coverage verification results obtained in step D343 are that the test coverage of the variant test code is less than the coverage threshold, then the process can return to step 150. In addition, if among all the coverage verification results obtained in step D343, only one coverage verification result is that the test coverage of the variant test code is greater than or equal to the coverage threshold, then the variant test code corresponding to the coverage verification result can be determined as the target test case.

[0119] If in step D343 there are at least two coverage verification results showing that the test coverage of the variant test code is greater than or equal to the coverage threshold, then the corresponding at least two variant test codes can be determined as target test cases; alternatively, based on the obtained coverage verification results, all variant test codes whose test coverage is greater than or equal to the coverage threshold are determined, and then all variant test codes whose test coverage is greater than or equal to the coverage threshold are compared in size, and the variant test code with the largest test coverage is determined as the target test case.

[0120] In some embodiments, the generating method further comprises:

[0121] E1. Obtain original test data, where the original test data is the first test case, the second test case, or the variant test code;

[0122] E2. Compile and repair the original test data using the language model to obtain target test data.

[0123] In the embodiment of the present application, if the compilation and running results in step C1 are all compilation failure types, steps E1 to E2 can be executed at this time, and the original test data in step E1 can be the first test case or the second test case, and the target test data in step E2 is the first test case after repair or the second test case after repair; or, if all the compilation verification results obtained in step D342 are mutation test code compilation failures, steps E1 to E2 can also be executed at this time, and the original test data in step E1 is the mutation test code, and the target test data is the repaired mutation test code. The compilation repair in step E2 is used to instruct the language model to repair the original test data that failed to compile, so as to obtain successfully compiled test data (i.e., target test data).

[0124] Furthermore, the target test data in step E2 is obtained by the following steps:

[0125] E21, obtain the invalid optimization threshold;

[0126] E22. Obtain the current invalid optimization count and first test data, as well as compilation error information corresponding to the first test data and a second code context, where the second code context is used to represent a code context related to the compilation error information;

[0127] E23. Constructing a repair prompt according to the compilation error information and the second code context, wherein the repair prompts corresponding to different loop rounds are different;

[0128] E24, inputting the first test data and the repair prompt into the language model to obtain second test data;

[0129] E25. Optimize and verify the second test data according to the first test data to obtain an optimization verification result;

[0130] E26. If the optimization verification result is that there is no compilation error information for the second test data, the second test data is determined as the target test data; or, if the optimization verification result is that the number of compilation error information of the second test data is less than the number of compilation error information of the first test data, the first test data is updated according to the second test data, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned; or, if the optimization verification result is that the number of compilation error information of the second test data is greater than or equal to the number of compilation error information of the first test data and the current number of invalid optimizations is less than the invalid optimization threshold, the current number of invalid optimizations is updated, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned.

[0131] In an embodiment of the present application, the invalid optimization threshold is used to indicate the maximum number of invalid optimizations that the language model performs on the original test data for compilation and repair. The specific value of the invalid optimization threshold can be any one of 2, 4, 7, 15, etc.; the first test data when the loop round is equal to 1 can be the original test data, and the first test data when the loop round is greater than or equal to 2 can be the first test data of the previous loop round or the second test data of the previous loop round.

[0132] It is understandable that the compilation error information may be the error information of the first test data during compilation, and the second code context may be an additional code context related to the compilation error information. Specifically, if the first test data of the current loop round is different from the first test data of the previous loop round, step E23 can be directly constructed by the error information of the first test data during compilation, such as at least one of the error type, bug location, bug element, etc., and the second code context related to the error information; or, if the first test data of the current loop round is the same as the first test data of the previous loop round, step E23 can be based on the repair prompt of the previous loop round, by adding a new error message to update the repair prompt of the previous loop round to obtain the repair prompt of the current loop round, so that the repair prompts corresponding to different loop rounds are different, thereby avoiding the situation where the language model has the same output when the same input is given.

[0133] It should be noted that the repair prompt of step E24 can be "You are a professional Java test engineer. Please generate a correct test case for {focus method name} based on the above compilation error information and code context, and output the complete test code", and the second test data can be the repaired test data output by the language model in the current loop round. For a certain loop round, the optimization verification in step E25 is used to verify whether the second test data output by the language model is a valid optimization, which can be determined by comparing the number of compilation errors of the second test data with the number of compilation errors of the first test data.

[0134] Specifically, if the optimization verification result is that there is no compilation error in the second test data, the second test data is successfully compiled, and the second test data output by the language model is effectively optimized, and the second test data can be directly determined as the target test data; or, if the optimization verification result is that the number of compilation error information of the second test data is less than the number of compilation error information of the first test data, the second test data output by the language model is also effectively optimized, and the second test data can be used as the first test data of the next cycle and return to step E22; or, if the optimization verification result is that the number of compilation error information of the second test data is greater than or equal to the number of compilation error information of the first test data and the current number of invalid optimizations is less than the invalid optimization threshold, it means that the second test data output by the language model is invalid optimization, and the first test data of the current cycle can be used as the first test data of the next cycle and return to step E22; or, if the optimization verification result is that the number of compilation error information of the second test data is greater than or equal to the number of compilation error information of the first test data and the current number of invalid optimizations is greater than or equal to the invalid optimization threshold, the compilation repair cycle can be exited at this time, and the language model is instructed to output the reason for the test failure and possible solutions. It should be added that the embodiment of the present application can effectively correct the compilation errors existing in the test cases generated by the language model through the cyclic process of compilation and repair, and can effectively improve the correctness of the test case generation.

[0135] A test case generation system proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0136] Reference Figure 2 , a test case generation system proposed in an embodiment of the present application includes:

[0137] An acquisition unit 101 is used to acquire a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to indicate the generation of a code intention corresponding to the first code context, and the first prompt is used to indicate the generation of a test case corresponding to the first code context;

[0138] A first processing unit 102 is used to input the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case;

[0139] A second processing unit 103 is used to input the first code context, the first code intent, and the first prompt into the language model based on different sampling temperatures to obtain a plurality of second test cases;

[0140] A third processing unit 104 is used to obtain a third test case according to the first test case and all the second test cases, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases;

[0141] The fourth processing unit 105 is used to input the third test case and the second prompt into the language model to obtain a target test case, wherein the second prompt is used to instruct generation of a test code corresponding to the third test case.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] Reference Figure 3 , the embodiment of the present application further provides an electronic device, including:

[0144] at least one processor 201;

[0145] At least one memory 202, used to store at least one program;

[0146] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0147] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0149] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0151] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0152] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0153] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0154] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0155] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0156] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0157] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0158] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A test case generation method, characterized in that: include: Acquire a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to instruct generation of a code intention corresponding to the first code context, and the first prompt is used to instruct generation of a test case corresponding to the first code context; Inputting the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case; Based on different sampling temperatures, the first code context, the first code intent, and the first prompt are input into the language model to obtain a plurality of second test cases; According to the first test case and all the second test cases, a third test case is obtained, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases; The third test case and the second prompt are input into the language model to obtain a target test case, wherein the second prompt is used to instruct generation of a test code corresponding to the third test case.

2. The generation method according to claim 1, characterized in that: The step of inputting the first code context, the intention instruction, and the first prompt into the language model to obtain a first code intention and a first test case corresponding to the first code context includes: Inputting the first code context and the intention instruction into the language model to obtain the first code intention, where the first code intention is used to indicate the code intention for generating the first code context; The first code context, the first code intent, and the first prompt are input into the language model to obtain the first test case.

3. The generation method according to claim 1, characterized in that: The first code context, the first code intent, and the first prompt are input into the language model based on different sampling temperatures to obtain a plurality of second test cases, including: Constructing an initial prompt according to the first code context, the first code intention and the first prompt; Based on different sampling temperatures, the initial prompt is sampled and updated by the language model to obtain a candidate prompt sequence, wherein the candidate prompt sequence includes a plurality of candidate prompts, each candidate prompt includes a first code context, a second code intent, and a third prompt word; The first code context, the corresponding second code intention and the corresponding third prompt are input into the language model to obtain a plurality of the second test cases.

4. The generation method according to claim 1, characterized in that: The step of obtaining a third test case according to the first test case and all the second test cases includes: Obtaining compilation and running results according to the first test case and all the second test cases; According to the compilation and running result, performing a first screening on the first test case and all the second test cases to obtain a number of successfully compiled test cases; The third test case is determined according to the sampling temperature corresponding to each of the successfully compiled test cases.

5. The generation method according to claim 1, characterized in that: The step of inputting the third test case and the second prompt into the language model to obtain a target test case includes: According to the second prompt, acquiring a semantic strategy and a variation strategy; Inputting the semantic strategy and the third test case into the language model to obtain a first test input sequence; According to the mutation strategy, the first test input sequence is mutated to obtain the target test case.

6. The generation method according to claim 5, characterized in that: The generating method further comprises: Get the preset time threshold; Get the current loop time and the second test input sequence; According to the mutation strategy, randomly mutate the second test input sequence to obtain a third test input sequence, wherein the number of test inputs in the second test input sequence of the current cycle round is greater than the number of test inputs in the second test input sequence of the previous cycle round; If the current cycle time is greater than or equal to the time threshold, the target test case is obtained according to the third test input sequence; or, if the current cycle time is less than the time threshold, the current cycle time is updated, and the second test input sequence is updated according to the third test input sequence, and then the step of obtaining the current cycle time and the second test input sequence is returned.

7. The generation method according to claim 6, characterized in that: The step of obtaining a target test case according to the third test input sequence comprises: Determining a variation test code according to the third test input sequence; Compile and verify the mutation test code to obtain a compilation verification result; If the compilation verification result is that the variation test code is compiled successfully, coverage verification is performed on the variation test code according to a preset coverage threshold to obtain a coverage verification result; If the coverage verification result is that the test coverage of the variant test code is greater than or equal to the coverage threshold, the variant test code is determined as the target test case.

8. The generation method according to claim 7, characterized in that: The generating method further comprises: Acquire original test data, where the original test data is the first test case, the second test case, or the variant test code; The original test data is compiled and repaired using the language model to obtain target test data.

9. The generation method according to claim 8, characterized in that: The target test data is obtained by the following steps: Get the invalid optimization threshold; Acquire the current invalid optimization count and first test data, as well as compilation error information corresponding to the first test data and a second code context, where the second code context is used to represent a code context related to the compilation error information; Constructing a repair prompt according to the compilation error information and the second code context, wherein the repair prompts corresponding to different loop rounds are different; Inputting the first test data and the repair prompt into the language model to obtain second test data; According to the first test data, optimizing and verifying the second test data to obtain an optimization and verification result; If the optimization verification result is that there is no compilation error information for the second test data, the second test data is determined as the target test data; or, if the optimization verification result is that the number of compilation error information of the second test data is less than the number of compilation error information of the first test data, the first test data is updated according to the second test data, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned; or, if the optimization verification result is that the number of compilation error information of the second test data is greater than or equal to the number of compilation error information of the first test data and the current number of invalid optimizations is less than the invalid optimization threshold, the current number of invalid optimizations is updated, and then the step of obtaining the current number of invalid optimizations and the first test data, and the compilation error information and the second code context corresponding to the first test data is returned.

10. A test case generation system, characterized in that: include: an acquisition unit, configured to acquire a first code context, an intention instruction corresponding to the first code context, a first prompt, and a language model, wherein the intention instruction is used to instruct generation of a code intention corresponding to the first code context, and the first prompt is used to instruct generation of a test case corresponding to the first code context; A first processing unit, configured to input the first code context, the intention instruction and the first prompt into the language model to obtain a first code intention and a first test case; A second processing unit is configured to input the first code context, the first code intent, and the first prompt into the language model based on different sampling temperatures to obtain a plurality of second test cases; A third processing unit, configured to obtain a third test case according to the first test case and all the second test cases, wherein the third test case is used to characterize a test case that is successfully compiled and run and has the lowest sampling temperature among all the test cases; The fourth processing unit is used to input the third test case and the second prompt into the language model to obtain a target test case, wherein the second prompt is used to instruct the generation of a test code corresponding to the third test case.

Images