A mock-aware unit test evaluation method for large models, equipment and medium

Abstract

The application discloses a large model-oriented Mock perception unit test evaluation method and device and a medium. The method comprises the following steps: constructing a Mock perception evaluation benchmark data set, extracting context information of a tested function and dividing the context information into three levels of configuration context; driving a large model to complete a Mock necessity detection task to obtain a Mock necessity prediction result, complete a Mock type classification task to obtain a Mock type prediction result, and complete a Mock perception unit test generation task to generate a test case; calculating multi-dimensional evaluation index values and aggregating the multi-dimensional evaluation index values to obtain an aggregation result and form a failure case classification statistical result; and integrating the aggregation result and the failure case classification statistical result to generate a comprehensive evaluation result. The application combines Mock perception benchmark construction, hierarchical context design and three-task quantitative evaluation, solves the problem that existing evaluation only focuses on surface code synthesis and ignores the correctness of the interaction level and lacks Mock-related capability special evaluation, and improves the comprehensiveness and fineness of the capability evaluation of the large model in the interaction perception test scene.

Description

Technical Field

[0001] This invention relates to the fields of software testing automation and artificial intelligence evaluation technology, and in particular to a method, device and medium for mock-aware unit testing evaluation of large models. Background Technology

[0002] In modern software engineering, unit testing is a fundamental means of ensuring the correctness and reliable evolution of software functionality. However, real-world software systems rarely operate in complete isolation; they often need to interact with external components such as databases, network services, file systems, and third-party application programming interfaces (APIs). To isolate these external dependencies, control side effects of the system under test, and verify the interaction semantics between components, developers typically use test surrogates such as Mock objects, Stub objects, Fake objects, and Spy objects. Among these, Mock objects, which can both simulate external behavior and explicitly assert interaction expectations, are the core carrier of interaction-level testing.

[0003] With the rapid development of large-scale models in code understanding and generation, automated unit test generation has gradually become a key path to improve testing efficiency. However, existing evaluation methods for large-scale models in unit testing tasks have the following shortcomings: First, existing benchmarks and evaluation protocols primarily focus on state assertion-based testing and input generation, neglecting interaction-level mock usage, mock type selection, and mock assertion generation. This makes it difficult for evaluations to reflect the core requirements of real-world software testing scenarios. Second, while some studies involve mock-related recommendations or synthesis, they typically focus only on a single segment of the mock usage process, lacking end-to-end systematic evaluation of mock-related capabilities. Third, existing evaluations mostly employ a single context approach, failing to systematically compare contexts of varying richness, such as class scope, import level, and inter-procedural contexts, making it difficult to reveal the true impact of context on the mock-related reasoning capabilities of large models. Fourth, existing evaluation metrics largely concentrate on syntactic correctness and compilation pass rate, lacking quantitative methods for deeper metrics such as mock configuration correctness and semantic consistency of interactive assertions, thus failing to truly reflect the upper limit of large models' capabilities in mock-aware testing tasks.

[0004] In summary, there is an urgent need in this field for a systematic evaluation method that considers interaction hierarchy, context, correctness of Mock configuration, and semantic consistency of interaction assertions. Summary of the Invention

[0005] This invention provides a method, device, and medium for mock-aware unit testing evaluation of large models, which solves the problems of existing evaluations neglecting the correctness of interaction levels, lacking systematic evaluation of mock-related capabilities, and insufficient analysis of context and failure modes, thereby improving the comprehensiveness, objectivity, and precision of large models' capability evaluation in mock-aware testing tasks.

[0006] This application provides a mock-aware unit testing evaluation method for large models, including the following steps: Based on the filtering rules, high-quality code repositories and test function data samples of mock statements are obtained. Then, the tested functions are matched to construct a dataset containing mock and non-mock statements and a mock-aware evaluation benchmark dataset. For each sample in the Mock-aware evaluation benchmark dataset, extract the context information of the function under test, and divide the context information into three levels of configuration context according to the richness of the context. Based on the Mock-aware evaluation benchmark dataset and its three-level configuration context, the large model to be evaluated is driven to complete the Mock necessity detection task and obtain the Mock necessity prediction result. Drive the large model to be evaluated to complete the Mock type classification task and obtain the Mock type prediction result; Drive the large model to be evaluated to complete the Mock-aware unit test generation task and obtain the generated test cases; A binary evaluation index is calculated for the Mock necessity detection results, a multi-category evaluation index is calculated for the Mock type classification results, and the syntax correctness, compilation pass rate, and semantic structure similarity are calculated for the generated test cases. The results are then integrated to obtain multi-dimensional evaluation index values. The multi-dimensional evaluation index values ​​are aggregated and analyzed according to the task dimension, context dimension, and model dimension to obtain the aggregated results. Failure cases are then categorized and statistically analyzed according to error categories to form failure case classification statistics. Finally, the aggregated results and the failure case classification statistics are integrated to generate the comprehensive evaluation results of the large model.

[0007] Furthermore, the Mock-aware evaluation benchmark dataset includes Mock-type test samples and non-Mock-type test samples. Each sample contains the function under test, the unit test cases written by the developer corresponding to the function under test, and the Mock type label. The specific screening rules are as follows: Code repositories for the target programming language are screened from the open-source code hosting platform; repositories are sorted by the number of stars to retain highly active repositories, while inactive repositories that have not been updated within the past year are removed; dependency installation and local build verification are performed on each repository to remove projects that cannot be compiled or run; and tutorial, assignment, and demo projects are removed based on the repository description. For each high-quality code repository, all test files are traversed according to the naming conventions of test functions, and an abstract syntax tree parsing tool is used to extract test functions that conform to the naming prefix or suffix conventions. For each test function, it is determined whether it is a mock test based on whether its function body contains mock-related application interfaces and mock-related construction patterns, and the test function body is required to contain at least one assertion statement to remove empty or placeholder tests, resulting in a sample of test function data containing mock statements. The specific process of matching the tested function is as follows: For each test function data sample containing a Mock statement, a combination of a heuristic strategy based on function naming and a structured strategy based on call chain analysis is used to identify the corresponding tested function, and then the test function and the tested function are matched to output a dataset containing test function and tested function data pairs. The heuristic strategy based on function naming and the structured strategy based on call chain analysis are as follows: First, the test function name is matched with the project-level function index after removing the test prefix; when the name matching fails, all non-assertion, non-Mock library function call chains called in the test function body are further parsed, and the method with the highest frequency in the project-level index is selected as the tested function. The specific steps for constructing a dataset containing both mock and non-mock statements are as follows: For each test function and test function in the dataset containing test function and test function data pairs, the source file containing the test function is analyzed using a static analysis tool to obtain the abstract syntax tree of the file. All test functions are identified from the abstract syntax tree, and test functions with non-mock statements related to the test function are extracted and merged with the test functions with mock statements to obtain a dataset containing both mock and non-mock statements. The dataset containing both mock and non-mock statements is manually classified and validated for consistency to form the mock-aware evaluation benchmark dataset. Specifically, the manual classification and consistency validation involves at least two annotators independently labeling the dataset containing both mock and non-mock statements according to preset classification rules, and then verifying the consistency of the labeling results. For data samples with discrepancies in labeling, the final category is determined after review, thus forming the mock-aware evaluation benchmark dataset.

[0008] Furthermore, the three-level configuration context specifically includes: class scope context, import enhancement context, and dependency enhancement context; The class scope context only includes the definition of the function being tested and the class name, attributes, field definitions and implementations of the class it belongs to, enabling the large model to perform reasoning based on the structured relationships within the class; The import enhancement context is above the class scope context and further includes all import statements related to the function under test in the file where the function under test is located; The dependency enhancement context, above the import enhancement context, further includes the source code definitions of methods directly called by the function under test and the methods that directly call the function under test, to provide an inter-procedural dependency view.

[0009] Furthermore, the specific steps for driving the large model under evaluation to complete the Mock necessity detection task are as follows: when the model writes unit tests for the function under test, a judgment criterion is introduced to determine whether Mock is needed, and a binary result is obtained, i.e., 1 indicates that Mock is needed and 0 indicates that Mock is not needed; the judgment criterion includes whether there are external dependencies, whether there are collaborators that are difficult to control, and whether network, database, or file input / output is involved; Based on the comparison between the binary results and the Mock type labels of the samples in the Mock perception evaluation benchmark dataset, the accuracy, precision, recall and F1 score are calculated. The binary results, accuracy, precision, recall and F1 score are then integrated to obtain the Mock necessity prediction result.

[0010] Furthermore, the Mock necessity detection task specifically involves: guiding the large model to determine which type of Mock should be used if the tested function is to be tested, and using a string matching method to extract the Mock category from the output of the large model as the Mock type prediction result; Based on the comparison between the Mock type prediction results and the Mock categories in the Mock type set of the Mock perception evaluation benchmark dataset, the micro-average F1 score, macro-average F1 score, and weighted F1 score are calculated; the Mock type set includes: database or object relational mapping class, file or input / output class, internal logic class, machine learning or data class, network or hypertext transfer protocol class, and system or process class.

[0011] Furthermore, the Mock-aware unit test generation task specifically involves: based on samples containing Mock statements in the Mock-aware evaluation benchmark dataset, the large model is required to directly generate test cases containing Mock configurations and interactive assertions, given the function under test and its context.

[0012] Furthermore, the grammatical correctness specifically involves parsing the generated code using a parser of the target language and statistically analyzing the proportion of samples that can be successfully parsed; the compilation pass rate specifically involves executing compilation in the native build environment of the target project and statistically analyzing the proportion of samples that can be successfully compiled; and the semantic structure similarity specifically involves quantifying the semantic alignment between the generated test cases and the test cases written by the developer based on abstract syntax tree parsing, from three dimensions: grammatical structure matching, data flow matching, and keyword matching.

[0013] Furthermore, the aggregation analysis of the multi-dimensional evaluation index values ​​by task dimension, context dimension, and model dimension specifically involves: performing a first-level aggregation of the multi-dimensional evaluation index values ​​by task type, and then aggregating the task-dimensional aggregation results by Mock necessity detection, Mock type classification, and Mock-aware unit test generation tasks respectively. The multi-dimensional evaluation index values ​​are aggregated at the second level according to the three-level configuration of class scope context, import enhancement context, and dependency enhancement context to obtain the aggregation result of context dimension; The multi-dimensional evaluation index values ​​are aggregated in the third layer according to the model category to distinguish the capability differences between commercial large models and open source large models, and obtain the aggregation results of the model dimensions. The specific steps for classifying and statistically analyzing failed cases by error category are as follows: the generated test functions are compiled using a code compiler, and samples that fail to compile are classified and statistically analyzed using a combination of manual open card classification and automatic clustering to obtain the classification and statistical results of failed cases. The categories for classification and statistical analysis include excessive mocking, false dependency inference, type misclassification, and oversensitivity to naming heuristics in the Mock necessity detection and Mock type classification tasks, and include two main categories, syntax errors and compilation errors, in the Mock-aware unit test generation task.

[0014] Compared with the prior art, the beneficial effects of the technical solution of this application include at least the following: First, by constructing a mock-aware evaluation benchmark dataset that includes both mock and non-mock samples and covers multiple real software domains, and by ensuring data quality through a multi-stage process including repository quality screening, time window constraints, naming conventions and call chain analysis, assertion statement filtering, and manual classification, we have solved the problems of existing evaluation datasets being small in scale, lacking mock guidance, having poor executability, and having a high risk of pre-training data leakage.

[0015] Second, through a three-level incremental context design of class scope context, imported enhanced context, and dependent enhanced context, a systematic analysis of the impact of context richness on the inference ability of large model mock is realized, which solves the problem that existing evaluations only use a single context form and cannot reveal the differences in the model's sensitivity to context.

[0016] Third, by dividing the Mock-related capabilities into three tasks—Mock necessity detection, Mock type classification, and Mock-aware unit test generation—and designing an evaluation index system that matches each task, end-to-end, fine-grained evaluation of the Mock-related capabilities of large models is achieved, solving the problems of fragmented task coverage and lack of overall coherence in existing evaluation methods.

[0017] Fourth, by introducing a semantic structure similarity index based on abstract syntax trees, the generated test cases are quantitatively scored from three dimensions: syntax structure, data flow, and keywords. This solves the problem that relying solely on syntax correctness and compilation pass rate is insufficient to reflect the deep-seated differences in generated quality.

[0018] Fifth, by categorizing and statistically analyzing failure cases according to error types, including excessive mocking, false dependency inference, type misclassification, overly sensitive naming heuristics, unclosed string literals, unclosed structural constructs, excessively nested mock constructs, duplicate patch parameter errors, invalid control flow or scope usage, and context execution mismatch, it provides diagnostic information that can guide the improvement of large models and testing frameworks, solving the problem that existing evaluations only provide overall indicators but lack action guidelines. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall process of the Mock-aware unit testing and evaluation method for large models in this invention; Figure 2 This is a schematic diagram illustrating the construction process of the Mock perception evaluation benchmark dataset in this invention; Figure 3 This is a schematic diagram of the assembly relationship of the three-level incremental context configuration in this invention; Figure 4 This is a structural diagram of the electronic device provided by the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention addresses the problems of existing evaluation methods that only focus on surface-level code synthesis, neglect the correctness of interaction layers, and lack specific assessment of Mock-related capabilities by constructing a Mock-aware evaluation benchmark dataset, designing a three-level incremental context configuration, defining Mock necessity detection, and classifying Mock types and generating Mock-aware unit tests. It also supplements these with fine-grained classification analysis of failure cases. This improves the comprehensiveness, objectivity, and precision of large-scale model capability assessment on Mock-aware unit test tasks.

[0022] For ease of understanding, the specific process of this invention will be described below.

[0023] Firstly, this invention provides a mock-aware unit testing evaluation method for large models, and the unit test code generation algorithm framework diagram is as follows: Figure 1 As shown, it includes the following steps: Step 1: Construct a Mock-based benchmark dataset for perception evaluation.

[0024] The Mock-aware evaluation benchmark dataset includes Mock-type test samples and non-Mock-type test samples. Each sample includes the function under test, the unit test cases written by the developer corresponding to the function under test, context information, and a Mock type label.

[0025] like Figure 2 As shown, in one specific embodiment, step one includes the following steps: (a) Warehouse screening.

[0026] Code repositories for the target programming language (e.g., Python) are collected from the open-source code hosting platform GitHub, and high-quality code repositories are obtained based on filtering rules.

[0027] The specific selection rules are as follows: repositories are sorted by the number of stars to retain highly active repositories, while inactive repositories that have not been updated within the past year are removed. Dependency installation and local build verification are performed on each repository to remove projects that cannot be compiled or run. Furthermore, tutorial, assignment, and demo projects are removed based on their repository descriptions to avoid their testing lacking representativeness for real-world industrial scenarios. Simultaneously, to reduce the risk of data leakage from large model pre-training corpora, only test cases created or updated after a specified time point are retained, thus forming the high-quality code repository.

[0028] (b) Mock sample extraction.

[0029] Based on the high-quality code repository, a sample of test function data containing mock statements was obtained.

[0030] Specifically, in each repository of the high-quality code repository, all test files are traversed according to the naming convention of test functions, and the abstract syntax tree parsing tool is used to extract test functions that conform to the naming prefix or suffix convention of test functions. For each test function, it is determined whether it is a Mock test based on whether its function body contains Mock-related application interfaces and Mock-related construction patterns. The test function body is required to contain at least one assertion statement to eliminate empty tests or placeholder tests, and test function data samples containing Mock statements are obtained.

[0031] (c) The tested function is matched.

[0032] Based on the test function data samples containing Mock statements, a combination of a heuristic strategy based on function naming and a structured strategy based on call chain analysis is used to identify the corresponding function under test for each test function data sample. The test functions and the functions under test are then matched to output a dataset containing test function and function under test data pairs.

[0033] Specifically, the heuristic strategy based on function naming and the structured strategy based on call chain analysis are as follows: First, the test function name is matched with the project-level function index after removing the test prefix; when the name matching fails, the call chain of all non-assertion and non-Mock library functions called in the test function body is further parsed, and the method with the highest frequency in the project-level index is selected as the test function.

[0034] (d) Construct a dataset containing both mock and non-mock statements.

[0035] For each test function and test function in the dataset of test function and test function data pairs, the source file where both are located is located using string matching method. The abstract syntax tree of the source file where the test function is located is obtained by analyzing the source file using static analysis tool. All test functions are identified from the abstract syntax tree, and test functions that are not related to the test function and do not contain mock statements are extracted. These test functions are then merged with the test functions containing mock statements to construct a dataset that contains both mock and non-mock statements.

[0036] In real-world development, not all external calls require mocking; for lightweight scenarios such as local file reading and in-memory computation, using real dependencies better demonstrates test robustness. This sub-step enables the benchmark to cover both mocked and unmocked scenarios, providing complete monitoring signals for determining the necessity of mocking.

[0037] (e) Manual classification and consistency verification.

[0038] Based on the dataset containing both mock and non-mock statements, several personnel with software testing experience were brought in to perform mock type semantic category labeling on the samples using the open card classification method, and the correspondence between test cases and the functions under test was verified a second time. For samples with disagreements or uncertainties, senior developers arbitrated; consistency metrics were used to ensure the quality of the labeling, thus ultimately forming a high-quality mock-aware evaluation benchmark dataset.

[0039] Through the above multi-stage process, each sample in the constructed Mock-aware evaluation benchmark dataset contains a complete tested function, corresponding test function, context information, and Mock type label. Furthermore, the Mock-aware evaluation benchmark dataset covers multiple interaction domains, including database or object relational mapping, file or input / output, internal logic, machine learning or data, network or hypertext transfer protocol, and system or process. This lays a high-quality data foundation for subsequent multi-dimensional evaluation and solves the problems of existing evaluation datasets being small in scale, lacking Mock guidance, having poor executability, and having a high risk of pre-training data leakage.

[0040] Step 2: Extract the context information of the tested function for each sample in the Mock-aware evaluation benchmark dataset, and divide the context information into three levels of configuration context according to the richness of the context.

[0041] like Figure 3 As shown, the three-level configuration context provided by this invention is as follows: The first level is the class scope context, which only contains the definition of the function being tested, the class name of its class, the definitions of its properties and fields, and the implementations of other methods within the class. This class scope context supports mock-related inference based on the structured relationships within the class (i.e., method calls and field accesses within the class), but does not introduce external module information.

[0042] The second level is the import enhancement context, which further incorporates all import statements related to the function under test in the file containing the function under test, based on the class scope context. This allows the model to explicitly perceive the external modules, libraries, and third-party application interfaces that the method depends on, thereby providing key external dependency signals for determining whether Mock is needed.

[0043] The third level is the dependency enhancement context, which further incorporates the source code definitions of methods directly called by the function under test (i.e., the callee) and methods that directly call the function under test (i.e., the caller) on the basis of the import enhancement context. This provides a generalized view of inter-procedural execution flow and dependencies, enabling the model to reason about deep dependencies along the call chain.

[0044] At each level of context configuration, this invention employs a unified prompt template to structurally extract and assemble each field, ensuring that all large models to be evaluated and all samples use structurally consistent prompt input. This uniformity guarantees the comparability and reproducibility of evaluation results between different models and different samples.

[0045] Step 3: Drive the large model to be evaluated to complete the Mock necessity detection task and obtain the Mock necessity prediction results.

[0046] Specifically, the Mock necessity detection task is formalized as a binary classification problem: taking the Mock-aware evaluation benchmark dataset and its three-level configuration context as input, when the model writes unit tests for the function under test, a decision criterion is introduced to determine whether Mock needs to be used, and the Mock necessity prediction result is obtained.

[0047] The judgment criteria include: the existence of external dependencies, the existence of uncontrollable collaborators (such as network calls, database access, file input / output), and whether operations with side effects or time sensitivity are involved. The model is required to output only single-character binary results, i.e., 1 indicates mocking is needed, and 0 indicates mocking is not needed, and additional explanations are prohibited to avoid inconsistent output formats interfering with automated evaluation.

[0048] Based on the comparison between the binary results and the Mock type labels of the samples in the Mock perception evaluation benchmark dataset, accuracy, precision, recall, and F1 score are calculated according to a binary classification index system. Accuracy reflects the overall proportion of correct judgments; precision reflects the proportion of samples that the model predicts needing mocking but actually need mocking; recall reflects the proportion of samples that actually need mocking but are successfully identified by the model; and the F1 score is the harmonic mean of precision and recall, reflecting the model's overall performance on both positive and negative samples. These indicators comprehensively characterize the ability of a large model to determine the necessity of mocking.

[0049] The Mock necessity prediction results include the binary result, accuracy, precision, recall, and F1 score.

[0050] Step 4: Drive the large model to be evaluated to complete the Mock type classification task and obtain the Mock type prediction results.

[0051] The Mock type classification task is formalized as a multi-class classification problem: based on samples containing Mock statements in the Mock-aware evaluation benchmark dataset, output the micro-average F1 score, macro-average F1 score, and weighted F1 score.

[0052] Specifically, the three-level context configuration and the function under test described in step two are provided to the large model to guide the large model in determining which type of Mock should be used if the function under test is to be tested. The Mock category is extracted from the output of the large model using a string matching method as the Mock type prediction result.

[0053] Furthermore, this invention also compares the Mock type prediction results with the Mock categories in the Mock type set of the Mock perception evaluation benchmark dataset to calculate the micro-average F1 score, macro-average F1 score, and weighted F1 score. The Mock type set in this invention includes, but is not limited to: database or object-relational mapping classes, file or input / output classes, internal logic classes, machine learning or data classes, network or hypertext transfer protocol classes, and system or process classes. In addition to providing the tested function and context, the prompt template explicitly lists the above categories and their brief descriptions, requiring the model to output only a unique category label. The micro-average F1 score treats each sample equally, reflecting overall performance and is dominated by the category with the largest sample size; the macro-average F1 score treats each category equally, more sensitively reflecting the model's performance on a minority of classes; the weighted F1 score is a weighted average based on the F1 scores of each category according to the sample size, taking into account both overall performance and category distribution. Mock type classification is generally more difficult than Mock necessity detection because it requires the model to distinguish fine-grained dependency semantics and be robust to class imbalance.

[0054] Step 5: Drive the large model to be evaluated to complete the Mock-aware unit test generation task and obtain the generated test cases.

[0055] Based on the mock statements included in the mock-aware benchmark dataset, the large model is required to directly generate a complete test case containing mock configuration and interactive assertions, given the function under test and its context. The model needs to simultaneously perform three aspects of reasoning: dependency isolation (determining which dependencies need to be replaced with mocks), test surrogate selection (determining the mock type to be used for each replaced dependency), and interactive assertion generation (writing assertions for the number of calls to the mock object, call parameters, and other interactive behaviors). This task should reflect the model's comprehensive capabilities in mock-aware test generation end-to-end.

[0056] Step 6: Calculate binary evaluation metrics for the Mock necessity detection results, calculate multi-category evaluation metrics for the Mock type prediction results, calculate syntax correctness, compilation pass rate, and semantic structure similarity for the generated test cases, and integrate them to obtain multi-dimensional evaluation metric values.

[0057] Specifically, the grammatical correctness refers to parsing the generated code using a target language parser and calculating the percentage of samples that can be successfully parsed. The compilation pass rate refers to executing compilation in the target project's native build environment and calculating the percentage of samples that can successfully compile (including dependency resolution). The semantic structure similarity refers to quantifying the semantic alignment between generated test cases and developer-written test cases based on abstract syntax tree parsing, considering grammatical structure matching, data flow matching, and keyword matching, to obtain a comprehensive score. This comprehensive score, compared to pure grammatical or pure compilation metrics, can more meticulously depict deeper differences in generation quality.

[0058] During the metric calculation process, this invention consistently employs zero-temperature greedy decoding to ensure the reproducibility of the generated results. For locally deployed open-source large models, a high-performance inference service is used for loading; for commercial large models accessed through application programming interfaces (APIs), their official APIs are used uniformly, maintaining the same low-temperature parameters. Furthermore, this invention repeats the experiment at least three times for each task-context configuration combination to verify the stability and variance level of the results; when the variance is observed to significantly exceed a preset threshold, the corresponding model and configuration are retried.

[0059] Step 7: Aggregate and analyze the multi-dimensional evaluation index values ​​according to the task dimension, context dimension, and model dimension to obtain the aggregation result, and classify and statistically analyze the failure cases according to the error category to form the failure case classification statistics result; finally, integrate the aggregation result and the failure case classification statistics result to generate the comprehensive evaluation result of the large model.

[0060] like Figure 4 As shown, in one specific embodiment, step seven includes the following steps: (a) Task dimension aggregation.

[0061] The multi-dimensional evaluation index values ​​are aggregated according to Mock necessity detection, Mock type classification, and Mock-aware unit test generation tasks to obtain the aggregated results of the task dimensions, forming the overall performance profile of the model on each type of task.

[0062] (b) Context-based aggregation.

[0063] Under each type of task, the multi-dimensional evaluation index values ​​are grouped and aggregated according to the three-level configuration of class scope context, import enhancement context, and dependency enhancement context to obtain the aggregated result of context dimension, thereby quantifying the impact of context richness on model performance.

[0064] (c) Model dimension aggregation.

[0065] The multi-dimensional evaluation metrics are grouped and aggregated into two main categories: commercial large models and open-source large models. The differences in capabilities of large models from different sources, of different scales, and with different training paradigms on mock-related tasks are compared. Furthermore, within each category, models are grouped by parameter size to reveal the relationship between model size and mock-related capabilities, thus obtaining the aggregated results of the model dimensions.

[0066] (d) Categorization and statistics of failure cases.

[0067] The generated test functions are compiled using a code compiler. Samples that fail to compile are then classified and statistically analyzed using a combination of manual open card classification and automatic clustering to generate a classification and statistical result of failure cases.

[0068] For classification tasks, error categories include, but are not limited to: excessive mocking (incorrectly predicting the need for mocking or assigning a mock type), spurious dependency inference (inferring non-existent external dependencies based on weak signals), type misclassification (choosing the wrong mock type when mocking is required), and overly sensitive naming heuristics (associating a specific mock category solely with specific words). For generation tasks, error categories include, but are not limited to: unclosed string literals, unclosed structural constructs, general syntax errors, degenerate or empty generation, excessively nested mock constructs, duplicate patch parameter errors, invalid control flow or scope usage, and context execution mismatch.

[0069] (e) Structured report output.

[0070] The aggregated results of the above task dimension, context dimension, and model dimension, along with the statistical results of failure case classification, are integrated into a structured comprehensive evaluation result and presented in a visual manner.

[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the method described above can be referred to the corresponding process in the aforementioned method, and will not be repeated here.

[0072] like Figure 4 As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.

[0073] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0074] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0075] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0076] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0077] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0078] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0079] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A mock-aware unit testing evaluation method for large models, characterized in that, Includes the following steps: Based on the filtering rules, high-quality code repositories and test function data samples of mock statements are obtained. Then, the tested functions are matched to construct a dataset containing mock and non-mock statements and a mock-aware evaluation benchmark dataset. For each sample in the Mock-aware evaluation benchmark dataset, extract the context information of the function under test, and divide the context information into three levels of configuration context according to the richness of the context. Based on the Mock-aware evaluation benchmark dataset and its three-level configuration context, the large model to be evaluated is driven to complete the Mock necessity detection task and obtain the Mock necessity prediction result. Drive the large model to be evaluated to complete the Mock type classification task and obtain the Mock type prediction result; Drive the large model to be evaluated to complete the Mock-aware unit test generation task and obtain the generated test cases; A binary evaluation index is calculated for the Mock necessity detection results, a multi-category evaluation index is calculated for the Mock type classification results, and the syntax correctness, compilation pass rate, and semantic structure similarity are calculated for the generated test cases. The results are then integrated to obtain multi-dimensional evaluation index values. The multi-dimensional evaluation index values ​​are aggregated and analyzed according to the task dimension, context dimension, and model dimension to obtain the aggregated results. Failure cases are then categorized and statistically analyzed according to error categories to form failure case classification statistics. Finally, the aggregated results and the failure case classification statistics are integrated to generate the comprehensive evaluation results of the large model.

2. The method according to claim 1, characterized in that, The Mock-aware evaluation benchmark dataset includes Mock-type test samples and non-Mock-type test samples. Each sample contains the function under test, the unit test cases written by the developer corresponding to the function under test, and the Mock type label. The specific screening rules are as follows: Code repositories for the target programming language are screened from the open-source code hosting platform; repositories are sorted by the number of stars to retain highly active repositories, while inactive repositories that have not been updated within the past year are removed; dependency installation and local build verification are performed on each repository to remove projects that cannot be compiled or run; and tutorial, assignment, and demo projects are removed based on the repository description. For each high-quality code repository, all test files are traversed according to the naming conventions of test functions, and an abstract syntax tree parsing tool is used to extract test functions that conform to the naming prefix or suffix conventions. For each test function, it is determined whether it is a mock test based on whether its function body contains mock-related application interfaces and mock-related construction patterns, and the test function body is required to contain at least one assertion statement to remove empty or placeholder tests, resulting in a sample of test function data containing mock statements. The specific process of matching the tested function is as follows: For each test function data sample containing a Mock statement, a combination of a heuristic strategy based on function naming and a structured strategy based on call chain analysis is used to identify the corresponding tested function, and then the test function and the tested function are matched to output a dataset containing test function and tested function data pairs. The heuristic strategy based on function naming and the structured strategy based on call chain analysis are as follows: First, the test function name is matched with the project-level function index after removing the test prefix; when the name matching fails, all non-assertion, non-Mock library function call chains called in the test function body are further parsed, and the method with the highest frequency in the project-level index is selected as the tested function. The specific steps for constructing a dataset containing both mock and non-mock statements are as follows: For each test function and test function in the dataset containing test function and test function data pairs, the source file containing the test function is analyzed using a static analysis tool to obtain the abstract syntax tree of the file. All test functions are identified from the abstract syntax tree, and test functions with non-mock statements related to the test function are extracted and merged with the test functions with mock statements to obtain a dataset containing both mock and non-mock statements. The dataset containing both mock and non-mock statements is manually classified and validated for consistency to form the mock-aware evaluation benchmark dataset. Specifically, the manual classification and consistency validation involves at least two annotators independently labeling the dataset containing both mock and non-mock statements according to preset classification rules, and then verifying the consistency of the labeling results. For data samples with discrepancies in labeling, the final category is determined after review, thus forming the mock-aware evaluation benchmark dataset.

3. The method according to claim 1, characterized in that, The three-level configuration context specifically includes: class scope context, import enhancement context, and dependency enhancement context; The class scope context only includes the definition of the function being tested and the class name, attributes, field definitions and implementations of the class it belongs to, enabling the large model to perform reasoning based on the structured relationships within the class; The import enhancement context is above the class scope context and further includes all import statements related to the function under test in the file where the function under test is located; The dependency enhancement context, above the import enhancement context, further includes the source code definitions of methods directly called by the function under test and the methods that directly call the function under test, to provide an inter-procedural dependency view.

4. The method according to claim 1, characterized in that, The specific steps for driving the large model under evaluation to complete the Mock necessity detection task are as follows: when the model writes unit tests for the function under test, a judgment criterion is introduced to determine whether Mock is needed, and a binary result is obtained, that is, 1 indicates that Mock is needed and 0 indicates that Mock is not needed; the judgment criterion includes whether there are external dependencies, whether there are collaborators that are difficult to control, and whether network, database or file input and output are involved. Based on the comparison between the binary results and the Mock type labels of the samples in the Mock perception evaluation benchmark dataset, the accuracy, precision, recall and F1 score are calculated. The binary results, accuracy, precision, recall and F1 score are then integrated to obtain the Mock necessity prediction result.

5. The method according to claim 1, characterized in that, The specific task of Mock necessity detection is as follows: guide the large model to determine which type of Mock should be used if the tested function is to be tested, and use string matching method to extract the Mock category from the output of the large model as the Mock type prediction result; Based on the comparison between the Mock type prediction results and the Mock categories in the Mock type set of the Mock perception evaluation benchmark dataset, the micro-average F1 score, macro-average F1 score, and weighted F1 score are calculated; the Mock type set includes: database or object relational mapping class, file or input / output class, internal logic class, machine learning or data class, network or hypertext transfer protocol class, and system or process class.

6. The method according to claim 1, characterized in that, The specific task of generating the Mock-aware unit test is as follows: based on the samples containing Mock statements in the Mock-aware evaluation benchmark dataset, the large model is required to directly generate test cases containing Mock configurations and interactive assertions under the given test function and its context.

7. The method according to claim 1, characterized in that, The grammatical correctness is specifically achieved by parsing the generated code using the target language's syntax parser and statistically analyzing the proportion of samples that can be successfully parsed; the compilation pass rate is specifically achieved by executing compilation in the target project's native build environment and statistically analyzing the proportion of samples that can be successfully compiled; the semantic structure similarity is specifically achieved by quantifying the semantic alignment between the generated test cases and the test cases written by the developer based on abstract syntax tree parsing, from three dimensions: syntax structure matching, data flow matching, and keyword matching.

8. The method according to claim 1, characterized in that, The aggregation analysis of the multi-dimensional evaluation index values ​​by task dimension, context dimension, and model dimension specifically involves: performing a first-level aggregation of the multi-dimensional evaluation index values ​​by task type, and then aggregating by Mock necessity detection, Mock type classification, and Mock-aware unit test generation tasks to obtain the aggregation results for the task dimension. The multi-dimensional evaluation index values ​​are aggregated at the second level according to the three-level configuration of class scope context, import enhancement context, and dependency enhancement context to obtain the aggregation result of context dimension; The multi-dimensional evaluation index values ​​are aggregated in the third layer according to the model category to distinguish the capability differences between commercial large models and open source large models, and obtain the aggregation results of the model dimensions. The specific steps for classifying and statistically analyzing failed cases by error category are as follows: the generated test functions are compiled using a code compiler, and samples that fail to compile are classified and statistically analyzed using a combination of manual open card classification and automatic clustering to obtain the classification and statistical results of failed cases. The categories for classification and statistical analysis include excessive mocking, false dependency inference, type misclassification, and oversensitivity to naming heuristics in the Mock necessity detection and Mock type classification tasks, and include two main categories, syntax errors and compilation errors, in the Mock-aware unit test generation task.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the Mock-based unit testing and evaluation method for large models as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the Mock-aware unit testing and evaluation method for large models as described in any one of claims 1-8.

Images