High-reliability unit test automatic generation method and apparatus based on conversational large language model

Abstract

The present invention provides a high-reliability unit test automatic generation method and apparatus based on a conversational large language model. First, projects are parsed and analyzed to extract context information, then the testing necessity of a method to be tested is determined, and different test direction lists are generated on the basis of the division of testing purposes. After initial test cases are generated and repaired, input data searching is performed on passed test cases to improve test input data. Subsequently, mutation testing is performed on all test cases, and test cases are regenerated for failed mutations to improve assertion quality. Finally, successful test cases are reduced and merged to avoid overlapping. The present invention is superior to existing methods in terms of coverage, test data integrity, assertion quality, and the like, and has stable performance across projects of different scales and fields. The generated test cases have significant advantages in readability, understandability, maintainability, test input data integrity, assertion quality, and volume.

Description

High-reliability unit test automatic generation method and device based on conversational large language model Technical Field

[0001] The present invention belongs to the technical field of automatic unit test generation, and in particular relates to a method and device for automatic generation of highly reliable unit tests based on a conversational large language model. Background Art

[0002] As software becomes increasingly larger and more complex, the demand for software quality is also increasing. Even minor defects can cause significant losses to businesses. Therefore, software testing, as the final step in software delivery, is becoming increasingly important. Unit testing, as the largest and most fundamental component of the software testing pyramid, is the most crucial step in the entire testing process. However, writing unit test cases requires significant time and effort, and is therefore often overlooked by developers. The emergence of automatic unit test case generation technology aims to address this problem. Automatic unit test case generation involves automatically generating a set of unit test cases that covers as many code paths and logic as possible to ensure the correctness and stability of software units. Unit test cases generated using this technology can improve test coverage, reduce the workload of manual test case writing, and enable earlier detection and repair of software errors and defects.

[0003] Currently, existing unit test case generation methods can be divided into three categories: methods based on traditional program analysis, methods based on pre-trained models, and methods based on conversational large language models. Methods based on traditional program analysis typically use mutation testing, random testing, symbolic execution, and other techniques to generate unit test cases; methods based on pre-trained models use large-scale English and code corpora to train pre-trained models, and define the unit test case generation task as a translation task, directly translating the code to be tested into test cases; methods based on conversational large language models generate prompts by inputting the context information of the method to be tested into a preset prompt template, and then use conversation to let the large language model generate unit tests for the method to be tested. However, these methods have the following unresolved technical problems:

[0004] 1. The variable and function names of unit test cases generated based on traditional program analysis methods do not have any semantics. They only focus on how to achieve higher coverage. They have poor readability and unclear test intent, making them difficult for developers to understand and maintain.

[0005] 2. Unit test cases generated using pre-trained models have certain advantages in readability, but the coverage of target methods is generally low, meaning that the percentage of generated unit test cases that successfully call the target method is low. On the other hand, the percentage of successful unit test cases is also low, but a significant portion of these test cases can be fixed with simple steps. Pre-trained models lack interactive capabilities and can only be generated multiple times to compensate for this shortcoming. As a result, the generated test cases often have suboptimal coverage of the target methods.

[0006] 3. Existing methods based on conversational large language models are not highly reliable, often suffering from issues such as duplicate generation, incomplete test input data coverage, poor assertion quality, and low success rates for generated unit test cases. Repeatedly using conversational large language models to generate tests for the same method under test can lead to duplicate coverage, incomplete searches, and similarity in the generated tests, making it impossible to fundamentally guarantee the quality of unit tests.

[0007] Summary of the Invention

[0008] Based on the above background, the present invention proposes a highly reliable method for automatically generating unit tests based on a conversational large language model, which can parse the abstract syntax tree to obtain class-level context information, obtain method-level context information by performing code reachability analysis on the method to be tested, and then determine the necessity of testing the method to be tested. Different test directions are then divided according to different test purposes, initial test cases are generated according to different test directions, and attempts are made to repair test cases in which errors occur. The basic data types of successful test cases are then exposed, and then a search for test input data is performed to maximize the coverage of test input data. Existing test cases are then subjected to mutation testing, and failed test cases are regenerated to maximize the quality of assertions. Finally, the test suite consisting of all successful test cases is reduced to avoid generating test cases with repeated coverage.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] According to a first aspect of this specification, a method for automatically generating high-reliability unit tests based on a conversational large language model is provided, comprising the following steps:

[0011] S1: Parse the project to be tested, extract the object files, parse the object files, extract the list of methods to be tested from the class-level context information, perform code reachability analysis on each method to obtain method-level context information, including the dependent classes, fields, and calling methods used by each method to be tested;

[0012] S2: For each method to be tested, a conversational large language model is used to determine the necessity of testing. If the method to be tested is determined to be necessary, the test objectives are divided and a list of test directions is obtained.

[0013] S3, starting from a test direction in the test direction list, uses dynamic context generation technology to generate a prompt containing as much context of the test method as possible within the maximum prompt length limit based on the code reachability analysis results of the test method. The prompt is then input into the conversational large language model to obtain an answer containing the initial test case.

[0014] S4, extracting an initial test case from the answer containing the initial test case, performing syntax verification on the initial test case, compiling and executing the test case, and if an error occurs during this process, attempting to fix the error using a heuristic method and / or a conversational large language model;

[0015] S5, for each successful test case, first expose the basic data type fields in the test case, and then search the space of test input data;

[0016] S6: For all generated test cases, mutation testing is used to verify the assertion quality of the test cases. For any mutations that are not successfully resolved, the mutation information is incorporated into a new prompt, and new test cases are generated again using the conversational large language model until the mutation is resolved or the maximum number of attempts is reached.

[0017] S7, minimize the test suite consisting of all successfully generated test cases to ensure that the test cases in the suite do not repeat in terms of test purpose, coverage, and mutation resolution.

[0018] Furthermore, S1 is specifically:

[0019] S1.1, traverse the entire project to be tested, extract the object files, and parse the object files. Use the parser to parse the object files into the form of abstract syntax tree AST;

[0020] S1.2, starting from the root node of the abstract syntax tree, traverse the entire abstract syntax tree and extract class-level context information from the abstract syntax tree, including package declaration statements, introduced external dependencies, class signatures, class constructors, class fields, getter and setter method signatures, and a list of methods to be tested;

[0021] S1.3, perform code reachability analysis for each method to be tested and obtain method-level context information, including the method body, method modifiers, whether the method uses fields, whether the method uses getters and setters, the dependent classes used by the method, and information about the calling method.

[0022] Furthermore, S2 is specifically:

[0023] S2.1: For each method to be tested, a prompt template is rendered that includes a test necessity judgment and test purpose division. A conversational large language model is input, and the answer format is specified. Regular expressions are used to determine test necessity based on the answer content.

[0024] S2.2, if the conversational large language model determines that the method to be tested needs to be tested, a list of test directions obtained by dividing the test objectives is obtained based on the answer content.

[0025] Furthermore, S3 is specifically:

[0026] S3.1: Based on a test direction, key information about the class of the method to be tested, and the reachable code blocks of the method to be tested, provide as much context as possible for the method to be tested, rendering it as a prompt while ensuring that the preset maximum prompt length is not exceeded. Specifically:

[0027] Attempt to generate a minimum context, which must include the method body, the signature of the class in which it is located, the signature of the constructor, the signatures of the reachable fields, and the signatures of the corresponding getters and setters. If the length of the minimum context is greater than the preset maximum hint length, the attempt to generate the test case will be canceled.

[0028] Dynamically add more contexts for the method to be tested based on whether the method to be tested depends on external classes. If the method to be tested depends on external classes, try to add the signature of the external class and the called method to the context information. After adding the information, a prompt will be generated to ensure that the length of the prompt is less than the preset maximum prompt length. If the method to be tested has no external dependencies, try to add the signature of the called method in the class where the method to be tested is located to the context information. If the preset maximum prompt length is still not exceeded, try to add the signatures of all methods in the class where the method to be tested is located to the context information. If the prompt exceeds the preset maximum prompt length when trying to add more information to the context information, the addition will be stopped.

[0029] S3.2, after generating the context of the method to be tested that meets the requirements in step S3.1, render the context into the corresponding prompt template according to the content of the context; specifically:

[0030] If the context contains dependency information, the context information is rendered into the template containing the dependency; if the context does not contain dependency information, the context information is rendered into the template without the dependency; finally, a prompt is generated that meets the maximum prompt length limit, conforms to the current test direction, and contains rich context of the method to be tested;

[0031] S3.3, input the prompt generated by S3.2 into the conversational large language model to obtain the answer containing the initial test case.

[0032] Furthermore, S4 is specifically:

[0033] S4.1, extract the initial test case. If the extraction fails, the attempt will be abandoned.

[0034] S4.2, perform syntax verification on the initial test case, specifically:

[0035] Use the syntax parser to try to parse valid test cases. If an error occurs during the parsing process, try to repair the test case using the statement end mark or code block end mark as a marker. If the repair fails, the attempt will be abandoned.

[0036] S4.3, verify the compilation correctness of the test case, specifically:

[0037] Use the compiler to try to compile the test case. If an error occurs during the compilation process, the extracted code snippet is compared line by line with the package declaration statement and import dependency statements of the class where the method to be tested is located. For the statements not included in the test case, the missing statements are added to the test case one by one. If a compilation error still occurs, the most relevant error message is matched according to the existing rules. The error message, test case, and context of the method to be tested are integrated to generate a prompt for fixing the error. If the repair fails, the attempt will be abandoned.

[0038] S4.4 After verifying the compilation correctness in S4.3, execute the test case; specifically:

[0039] If an error occurs during execution, the most relevant error information is matched using existing rules, and the error information, test case, and context of the method to be tested are integrated to generate prompts for fixing the error. If the repair fails, the attempt will be abandoned.

[0040] Furthermore, S4.1 is specifically:

[0041] There are two main forms of test cases in the answers of large language models: one is that there are clear delimiters, and the answer contains the test case between the start and end delimiters; the other is that there is no clear delimiter;

[0042] For the first type of answer, use regular expressions to match and extract all code snippets in the answer; filter all code snippets and select the code snippets containing the keywords "@Test", "class", and "import" as valid test cases;

[0043] For the second form of answer, the delimitation method is used to define the upper and lower bounds of the test case code snippet. Specifically, first locate the line containing the keywords "class" + method name + "Test", and use this line as the starting line to search for boundaries upward and downward respectively. In the process of searching for boundaries, determine whether the end character of a line is allowed to be the end character of a statement. If allowed, the line is determined to be part of the code snippet. Otherwise, the line is determined to be the boundary of the code snippet.

[0044] Furthermore, S5 is specifically:

[0045] S5.1. For each successfully passed test case, parse it and record the assignment statements of basic data types as search entry points.

[0046] S5.2, start the search process, enumerate and combine all possible values ​​of the basic data type, then modify the assignment statement in the original test case to form a new test case, execute the test case and record its execution results, including coverage and test pass status. Repeat the process until all possible values ​​have been traversed or the maximum search time limit is reached, at which point the search ends.

[0047] S5.3, based on the execution results during the search process, all test input data are classified, and finally only the test cases with different coverage and passed execution are retained.

[0048] Furthermore, S6 is specifically:

[0049] S6.1. Implant mutations into the original method to be tested, including condition mutations, arithmetic operator mutations, general expression mutations, statement mutations, boundary value mutations, return value mutations, and logical operator mutations, to form a mutated method to be tested;

[0050] S6.2: Run all test cases generated by all test directions on the mutated method under test. If a test case that originally passed fails on the mutated method under test, it means that the implanted mutation has been resolved. This step also requires recording the mutation resolution status.

[0051] S6.3: If there are still unresolved mutations, the mutation information and the context information of the method to be tested are dynamically rendered into the prompt template to generate a prompt to resolve the mutation. The prompt is input into the large language model to generate new test cases until the mutation is resolved or the maximum number of attempts is reached.

[0052] Furthermore, S7 is specifically:

[0053] S7.1 Based on the coverage of all generated test cases, select and delete the test cases with the largest volume, consistent coverage, and consistent mutation resolution, until all test cases have inconsistent coverage and mutation resolution.

[0054] S7.2 Merge all test cases into a test suite and run the test suite. If the test suite fails to run successfully, it means that there are conflicting test cases in the test suite. First, try adding steps to prepare and clean up the environment. If the conflict cannot be resolved successfully, try deleting some test cases.

[0055] According to a second aspect of this specification, a highly reliable unit test automatic generation device based on a conversational large language model is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the highly reliable unit test automatic generation method based on a conversational large language model as described in the first aspect.

[0056] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0057] 1. This paper proposes a highly reliable method for automatic unit test generation based on a conversational large language model. The method can automatically parse projects, perform code reachability analysis, extract relevant information, divide test objectives, dynamically generate context, automatically generate and repair initial test cases, then search the test input data of the initial test cases, use mutation testing to ensure the quality of the generated test cases, and finally minimize the test cases to ultimately output a correct, reliable, high-coverage, and compact test suite.

[0058] 2. In the experiment, the present invention has demonstrated stable results on different projects (including different characteristics such as size, field and version). Compared with traditional methods, it has shown significant advantages in both branch coverage and line coverage. Similarly, when compared with methods based on pre-trained models, the present invention is superior to such methods in both line coverage and coverage of the methods to be tested. Finally, when compared with existing methods based on conversational large language models, the present invention has shown significant advantages in terms of test intent, integrity of test input data, quality of test assertions, and repetitiveness of test cases.

[0059] 3. Experiments have shown that the test cases generated by the present invention have significant advantages in readability, comprehensibility, maintainability, test input data integrity, assertion quality, and volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of the process of generating a complete test suite.

[0061] FIG2 is a flowchart of parsing a project and extracting context information of the method to be tested.

[0062] FIG3 is a flowchart showing the division of test necessity and test direction.

[0063] FIG4 is a flow chart showing the process of generating initial test cases.

[0064] Figure 5 is a flowchart of verifying and repairing the initial test case.

[0065] Figure 6 is a schematic diagram of the test input data search process

[0066] FIG7 is a flow chart of the mutation test.

[0067] Figure 8 is a schematic diagram of the minimized test suite. DETAILED DESCRIPTION

[0068] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] As shown in FIG1 , an embodiment of the present invention provides a method for automatically generating highly reliable unit tests based on a conversational large language model, including the following steps:

[0070] S10: Parse the project to be tested, extract the object file, parse the object file, extract the list of methods to be tested from the class-level context information, perform code reachability analysis on each method to be tested, and obtain method-level context information, including the dependent classes, fields, and calling methods used by each method to be tested;

[0071] S20: For each method to be tested, the conversational large language model is used to determine the necessity of testing. If it is determined that the method to be tested is necessary, the test objectives are divided and a list of test directions is obtained;

[0072] S30, starting from a test direction in the test direction list, using a dynamic context generation technique based on the code reachability analysis results of the method to be tested, generating a prompt containing as much context of the method to be tested as possible within a maximum prompt length limit, and inputting the prompt into the conversational large language model to obtain an answer containing the initial test case;

[0073] S40, extracting an initial test case from the answer containing the initial test case, performing syntax verification on the initial test case, compiling and executing the test case, and if an error occurs during this process, attempting to fix the error using a heuristic method and / or a conversational large language model;

[0074] S50, for each successful test case, first expose the basic data type fields in the test case, and then search the space of test input data;

[0075] S60: For all generated test cases, mutation testing is used to verify the assertion quality of the test cases. For any mutations that are not successfully resolved, the mutation information is incorporated into a new prompt, and new test cases are generated again using the conversational large language model until the mutation is resolved or the maximum number of attempts is reached.

[0076] S70, minimizing the test suite composed of all successfully generated test cases to ensure that the test cases in the suite are not repeated in terms of test purpose, coverage, and mutation resolution.

[0077] As shown in Figure 2, the specific implementation of S10 is as follows:

[0078] S101, traverse the entire project to be tested, extract the object file, and parse the object file, using the parser to parse the object file into the form of an abstract syntax tree AST;

[0079] S102, starting from the root node of the abstract syntax tree, traversing the entire abstract syntax tree, and extracting class-level context information from the abstract syntax tree, including package declaration statements, introduced external dependencies, class signatures, class constructors, class fields, getter and setter method signatures, and a list of methods to be tested;

[0080] S103, perform code reachability analysis on each method to be tested, and obtain method-level context information, including the method body, method modifiers, whether the method uses fields, whether the method uses getters and setters, the dependent classes used by the method, and information about the calling method.

[0081] As shown in Figure 3, the specific implementation of S20 is as follows:

[0082] S201: For each method to be tested, a prompt template is rendered that includes test necessity judgment and test purpose classification. The conversational large language model is input and the answer format is specified. The test necessity is judged based on the answer content using regular expressions.

[0083] S202: If the conversational large language model determines that the method to be tested needs to be tested, a list of test directions divided by test objectives is obtained according to the answer content.

[0084] As shown in FIG4 , the specific implementation of S30 is as follows:

[0085] S301: Based on a test direction, key information of the class of the test method, and reachable code blocks of the test method, as much test method context as possible is provided and rendered as a prompt while ensuring that the preset maximum prompt length is not exceeded. Specifically:

[0086] First, try to generate a minimum context. The minimum context must include the method body, the signature of the class, the signature of the constructor, the signatures of the reachable fields, and the corresponding getter and setter signatures. If the length of the minimum context is greater than the preset maximum hint length, the attempt to generate the test case will be canceled.

[0087] Then, more contexts of the method to be tested are dynamically added based on whether the method to be tested depends on external classes; if the method to be tested depends on external classes, try to add the signature of the external class and the called method to the context information. After adding the information, a prompt will be generated to ensure that the length of the prompt is less than the preset maximum prompt length; if the method to be tested has no external dependencies, try to add the signature of the called method in the class where the method to be tested is located to the context information. If the preset maximum prompt length is still not exceeded, try to add the signatures of all methods in the class where the method to be tested is located to the context information; if the prompt exceeds the preset maximum prompt length when trying to add more information to the context information, the adding behavior will be stopped;

[0088] S302: After generating the context of the method to be tested that meets the requirements in step S301, the context is rendered into the corresponding prompt template according to the content of the context; specifically:

[0089] If the context contains dependency information, the context information is rendered into the template containing the dependency; if the context does not contain dependency information, the context information is rendered into the template without the dependency; finally, a prompt is generated that meets the maximum prompt length limit, conforms to the current test direction, and contains rich context of the method to be tested;

[0090] S303: Input the prompt generated in S302 into the conversational large language model to obtain an answer containing the initial test case.

[0091] As shown in FIG5 , the specific implementation of S40 is as follows:

[0092] S401: Extracting initial test cases. Test cases can exist in two main forms in the answers of the large language model: one with explicit delimiters, where the answer includes the test case within the start and end delimiters; the other without explicit delimiters.

[0093] For the first form of answer, the present invention uses regular expressions to match and extract all code snippets in the answer; since the extracted code snippets may be incomplete or do not contain test cases, the present invention further filters all code snippets and selects code snippets that contain the keywords "@Test", "class", and "import" as a valid test case.

[0094] In response to the second form of answer, the present invention adopts a delimitation method to define the upper and lower bounds of the test case code snippet; specifically, the present invention first locates the line containing the "class" + method name + "Test" keyword, and uses this line as the starting line to search for boundaries upward and downward respectively; specifically, in the process of searching for boundaries, it is determined whether the end characters of a line are allowed to be the end characters of a statement. These characters usually include ";", "}", ")", "{", "(", space, tab. If allowed, the line is determined to be part of the code snippet, otherwise the line is determined to be the boundary of the code snippet.

[0095] If the initial test case extraction fails, the attempt will be abandoned.

[0096] S402, after extracting valid initial test cases in S401, perform syntax verification on the initial test cases; specifically:

[0097] Use a syntax parser to try to parse a valid test case. If an error occurs during the parsing process, try to repair the test case using a statement terminator or a code block terminator as a marker; specifically, starting from the last character of the code snippet, search forward for the statement terminator ";" and the code block terminator "}". When the statement terminator or the code block terminator is found, stop, take out the previous code snippet, and try to add the character "}" to it to complete its structure. The present invention can ensure that the number of code block terminators matches the number of code block start symbols.

[0098] If the repair fails, the attempt will be abandoned.

[0099] S403: After verifying the grammatical correctness in S402, verify the compilation correctness of the test case; specifically:

[0100] Use the compiler to try to compile the test case. If an error occurs during the compilation process, the extracted code snippet is compared line by line with the package declaration statement and import dependency statement of the class where the method to be tested is located. For the statements not included in the test case, the missing statements are added to the test case one by one. If a compilation error still occurs, the most relevant error information is matched according to the existing rules, and the error information, test case, and context of the method to be tested are integrated to generate a prompt for fixing the error. Specifically, within the preset maximum prompt length limit, an attempt is made to add the test case, the context of the method to be tested, and the error information to render it into a prompt template for fixing the error to form a prompt. If the prompt length exceeds the maximum prompt length limit, an attempt is made to delete the error information until the restriction condition is met. At the same time, the present invention will also ensure the minimum error information length to ensure the effectiveness of the repair task.

[0101] If the repair fails, the attempt will be abandoned.

[0102] S404: After verifying the correctness of the compilation in S403, execute the test case; specifically:

[0103] If an error occurs during the execution process, the most relevant error information is matched according to the existing rules, and the error information, test case, and context of the method to be tested are integrated to generate a prompt for fixing the error. Specifically, within the preset maximum prompt length limit, an attempt is made to add the test case, context of the method to be tested, and error information to the prompt template for fixing the error to form a prompt. If the prompt length exceeds the maximum prompt length limit, an attempt is made to delete the error information until the restriction condition is met. At the same time, the present invention will also ensure the minimum error information length to ensure the effectiveness of the repair task.

[0104] If the repair fails, the attempt will be abandoned.

[0105] As shown in FIG6 , the specific implementation of S50 is as follows:

[0106] S501: For each successfully passed test case, first parse it and record the assignment statement of the basic data type as the search entry;

[0107] S502: Start the search process, enumerate and combine all possible values ​​of the basic data type, then modify the assignment statement in the original test case to form a new test case, execute the test case and record its execution results, including coverage and test pass status. Repeat the search until all possible values ​​have been traversed or the maximum search time limit is reached, at which point the search ends.

[0108] S503: Classify all test input data according to the execution results in the search process, and finally retain only test cases with different coverage and passed execution.

[0109] As shown in FIG7 , the specific implementation of S60 is as follows:

[0110] S601, implanting mutations into the original method to be tested, including condition mutations, arithmetic operator mutations, general expression mutations, statement mutations, boundary value mutations, return value mutations, and logical operator mutations, to form a mutated method to be tested;

[0111] S602: Run all test cases generated by all the aforementioned test directions on the mutated method under test. If the test cases that originally passed fail on the mutated method under test, it means that the implanted mutation has been resolved. This step also requires recording the mutation resolution status.

[0112] S603: If there are still unresolved mutations, the mutation information and the context information of the method to be tested are dynamically rendered into the prompt template to generate a prompt for resolving the mutation. The prompt is input into the large language model to generate new test cases until the mutation is resolved or the maximum number of attempts is reached.

[0113] As shown in FIG8 , the specific implementation of S70 is as follows:

[0114] S701, based on the coverage of all generated test cases, select and delete the test cases with the largest volume, consistent coverage, and consistent mutation resolution, until the coverage and mutation resolution of all test cases are inconsistent;

[0115] S702, merge all test cases into a test suite and run the test suite; if the test suite fails to run successfully, it means that there are conflicting test cases in the test suite. First try to add steps to prepare the environment and clean up the environment. If the conflict cannot be successfully resolved, try to delete some test cases.

[0116] Corresponding to the aforementioned embodiment of the method for automatically generating high-reliability unit tests based on a conversational large language model, the present invention also provides an embodiment of an apparatus for automatically generating high-reliability unit tests based on a conversational large language model.

[0117] An embodiment of the present invention provides a high-reliability unit test automatic generation device based on a conversational large language model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the high-reliability unit test automatic generation method based on a conversational large language model in the above embodiment.

[0118] The embodiments of the highly reliable automatic unit test generation apparatus based on a conversational large language model of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. The apparatus embodiments can be implemented using software, hardware, or a combination of software and hardware.

[0119] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0120] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0121] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for automatically generating high-reliability unit tests based on a conversational large language model in the above embodiment is implemented.

[0122] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0123] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A highly reliable unit test automatic generation method based on a conversational large language model, characterized in that: The following steps are involved: S1, parse the project to be tested, extract the object file, parse the object file, extract the list of methods to be tested from the class-level context information, perform code reachability analysis on each method to be tested, and obtain method-level context information, including the dependent classes, fields, and calling methods used by each method to be tested; S2: For each method to be tested, the conversational large language model is used to determine the necessity of testing. If it is determined that the method to be tested needs to be tested, the test objectives are divided and a list of test directions is obtained. S3, starting from a test direction in the test direction list, using the dynamic context generation technology to generate a prompt containing as much context of the method to be tested as possible within the maximum prompt length limit according to the code reachability analysis result of the method to be tested, and inputting the prompt into the conversational large language model to obtain an answer containing the initial test case; S4, extracting an initial test case from the answer containing the initial test case, performing syntax verification on the initial test case, compiling and executing the test case, and if an error occurs during the process, attempting to fix the error using a heuristic method and / or a conversational large language model; S5, for each successful test case, first expose the basic data type fields in the test case, and then search the space of test input data; S6: For all generated test cases, mutation testing is used to verify the assertion quality of the test cases. For mutations that are not successfully resolved, the mutation information is integrated into a new prompt, and new test cases are generated again using the conversational large language model until the mutation is resolved or the maximum number of attempts is reached. S7, minimize the test suite consisting of all successfully generated test cases to ensure that the test cases in the suite will not be repeated in terms of test purpose, coverage, and mutation resolution.

2. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S1 is specifically: S1.1, traverse the entire project to be tested, extract the object file, and parse the object file, using the parser to parse the object file into the form of an abstract syntax tree AST; S1.2, starting from the root node of the abstract syntax tree, traverse the entire abstract syntax tree and extract class-level context information from the abstract syntax tree, including package declaration statements, introduced external dependencies, class signatures, and class Method signatures of constructors, class fields, getters and setters, and a list of methods to be tested; S1.3, perform code reachability analysis for each method to be tested and obtain method-level context information, including the method body, method modifiers, whether the method uses fields, whether the method uses getters and setters, the dependent classes used by the method, and information about the calling method.

3. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S2 is specifically: S2.1, for each method to be tested, render it into a prompt template that includes test necessity judgment and test purpose division, input the conversational large language model, and specify the answer format, and use regular expressions to judge the test necessity based on the answer content; S2.2, if the conversational large language model determines that the method to be tested needs to be tested, a list of test directions divided by test purpose is obtained based on the answer content.

4. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S3 is specifically: S3.1, based on a direction to be tested, key information of the class to which the method to be tested belongs, and reachable code blocks of the method to be tested, provide as much context of the method to be tested as possible and render it into a prompt while ensuring that the preset maximum prompt length is not exceeded; Specifically: Try to generate a minimum context, which must include the method body, the signature of the class, the signature of the constructor, the reachable fields, and the signatures of the corresponding getters and setters; if the length of the minimum context is greater than the preset maximum hint length, the attempt to generate the test case will be canceled; Dynamically add more contexts of the method to be tested based on whether the method to be tested depends on external classes; if the method to be tested depends on external classes, try to add the signature of the external class and the called method in the context information. After adding the information, a prompt will be generated to ensure that the length of the prompt is less than the preset maximum prompt length; if the method to be tested has no external dependencies, try to add the signature of the called method in the class where the method to be tested is located in the context information. If it still does not exceed the preset maximum prompt length, try to add the signatures of all methods in the class where the method to be tested is located in the context information; if the prompt exceeds the preset maximum prompt length when trying to add more information to the context information, the adding behavior will be stopped; S3.2, after generating the context of the method to be tested that meets the requirements in step S3.1, rendering the context into the corresponding prompt template according to the content of the context; Specifically: If the context contains dependency information, render the context information to the template containing the dependency If the context does not contain any dependency information, the context information is rendered into a template without dependencies. Finally, a prompt is generated that meets the maximum prompt length limit, conforms to the current test direction, and is rich in the context of the method to be tested. S3.3, input the prompt generated by S3.2 into the conversational large language model to obtain an answer containing the initial test case.

5. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S4 is specifically: S4.1, extract the initial test case. If the extraction fails, the attempt will be abandoned; S4.2, perform syntax verification on the initial test case, specifically: Use the syntax parser to try to parse the valid test case. If an error occurs during the parsing process, try to repair the test case using the statement end mark or code block end mark as a marker. If the repair fails, the attempt will be abandoned. S4.3, verify the compilation correctness of the test case, specifically: Use the compiler to try to compile the test case. If an error occurs during the compilation process, compare the extracted code snippet with the package declaration statement and import dependency statements of the class where the method to be tested is located line by line. For the statements not included in the test case, add the missing statements to the test case one by one; If compilation errors still occur, the most relevant error information is matched through existing rules, and the error information, test cases, and context of the method to be tested are integrated to generate prompts for fixing the errors. If the repair fails, the attempt will be abandoned; S4.4 executes the test case after verifying the compilation correctness in S4.3; Specifically: If an error occurs during execution, the most relevant error information is matched through existing rules, and the error information, test cases, and context of the method to be tested are integrated to generate prompts for fixing the error. If the repair fails, the attempt will be abandoned.

6. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S4.1 specifically states: There are two main forms of test cases in the answers of large language models. One is that there are clear delimiters and the answer contains the test case between the start and end delimiters; the other is that there is no clear delimiter. For the first form of answer, use regular expressions to match and extract all code snippets in the answer; filter all code snippets to filter out those that contain "@Test", "class", "import" at the same time. The code snippet of the keyword serves as a valid test case; For the second form of answer, the delimitation method is used to define the upper and lower bounds of the test case code snippet. Specifically, first locate the line containing the keywords "class" + method name + "Test", and use this line as the starting line to search for boundaries upward and downward respectively. In the process of searching for boundaries, determine whether the end character of a line is allowed to be the end character of a statement. If allowed, determine that the line is part of the code snippet, otherwise determine that the line is the boundary of the code snippet.

7. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S5 is as follows: S5.1, for each successfully passed test case, parse it and record the assignment statements of basic data types as the search entry; S5.2, start the search process, enumerate and combine all possible values ​​of the basic data type, then modify the assignment statement in the original test case to form a new test case, execute the test case and record its execution results, including coverage and test pass status; repeat the execution until all possible values ​​have been traversed or the maximum search time limit is reached, and the search ends; S5.3, according to the execution results in the search process, all test input data are classified, and finally only the test cases with different coverage and passed execution are retained.

8. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S6 is specifically: S6.1, implant mutations into the original method to be tested, including condition mutations, arithmetic operator mutations, general expression mutations, statement mutations, boundary value mutations, return value mutations, and logical operator mutations, to form a mutated method to be tested; S6.2, run all test cases generated by all test directions on the mutated method to be tested. If the test cases that originally passed fail to execute on the mutated method to be tested, it means that the implanted mutation has been resolved. This step also needs to record the resolution of the mutation. S6.3, if there are still unresolved mutations, the mutation information and the context information of the method to be tested are dynamically rendered into the prompt template to generate a prompt for resolving the mutation, which is input into the large language model to generate new test cases until the mutation is resolved or the maximum number of attempts is reached.

9. The method for automatically generating high-reliability unit tests based on a conversational large language model according to claim 1, characterized in that: S7 is specifically: S7.1 For the coverage of all generated test cases, select and delete the test cases with the largest volume and coverage. Test cases that are consistent and solve the same mutations, until all test cases have inconsistent coverage and mutation solutions; S7.2 merge all test cases into one test suite and run the test suite; If the test suite fails to run successfully, it means that there are conflicting test cases in the test suite. First try to add steps to prepare and clean up the environment. If the conflict cannot be resolved successfully, try to delete some test cases.

10. A highly reliable unit test automatic generation device based on a conversational large language model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the high-reliability unit test automatic generation method based on a conversational large language model as described in any one of claims 1 to 9.

Images