Evolutionary testing system and method based on LLM and device

Abstract

This invention relates to an evolutionary testing system, method, and apparatus based on LLM (Liquidity Management Model), belonging to the field of computer technology. The evolutionary testing system includes: a multi-dimensional collaborative evaluation module; a targeted breakthrough evolution module configured to generate regular test cases, and when the update of the overall coverage result is deemed stagnant, to locate unreached paths based on the overall coverage result, extract logical constraints, obtain test cases whose execution trajectories are closest to the unreached paths, and infer root causes, thereby generating breakthrough test cases through targeted mutation; and a self-healing closed-loop module configured to execute test cases in the evolved population, capture error information generated during execution and iteratively repair it, storing repaired test cases with a comprehensive fitness exceeding a threshold in a test case library. This solves the problem that existing search methods struggle to penetrate deep code logic and achieve slow coverage improvement in automated testing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an evolutionary testing system, method and apparatus based on LLM. Background Technology

[0002] In the field of automated software test generation, efficiently generating test cases with high path coverage is a core challenge for ensuring software quality. Existing technologies mainly fall into two categories: one is search-based test generation methods, which use heuristic search methods such as genetic algorithms to iteratively evolve test inputs with code coverage as the fitness function; the other is test generation methods based on large language models, which utilize their code understanding capabilities to generate test cases.

[0003] However, the aforementioned technologies all have the following significant limitations: search-based methods rely on random mutation or predefined operators, lack a deep understanding of code semantics, and the mutation process is highly arbitrary, easily generating a large amount of invalid input, leading to a waste of computational resources. When tests enter deep logical branches, they are prone to getting stuck in local optima, resulting in slow or even stagnant coverage improvement; while large language models possess excellent code understanding and logical reasoning capabilities, they cannot autonomously establish explicit correlations with test coverage metrics, making it difficult to derive targeted evolutionary strategies based on the current coverage gap, and the generated test cases are often "compliant but useless." Existing methods that combine the two are mostly superficial, such as only calling the large model to generate new seeds when the search fails. The large model does not truly participate in the core decision-making process of the evolutionary algorithm and cannot systematically solve the problem of continuous coverage optimization. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an evolutionary testing system, method and apparatus based on LLM, which solves the technical problems of existing search-type technologies being blind and prone to local optima due to lack of semantic understanding, large language models being unable to autonomously associate coverage indicators and the shallow fusion of the two making it difficult to systematically solve the problem of continuous coverage optimization.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, embodiments of the present invention provide an evolutionary testing system based on LLM, comprising:

[0009] The semantic heuristic initialization module is configured to use RAG technology to retrieve reference code and test cases with similar structures to the code under test from the test case library. After integrating the retrieval results with the code under test and its contextual dependency information into prompt words, it is input into the first LLM to generate the initial population.

[0010] The multidimensional collaborative evaluation module is configured to execute each test case in the initial population to obtain path coverage and statistically analyze the overall coverage results, input each test case into the second LLM to obtain a semantic potential score, and calculate the comprehensive fitness of each test case based on the path coverage and semantic potential score.

[0011] The targeted breakthrough evolution module is configured to select test cases from the initial population, perform logical hybridization to generate regular test cases, and when the update of the overall coverage result is determined to be stagnant, locate the unreached path based on the overall coverage result, extract logical constraints, and obtain the test cases whose execution trajectories are closest to the unreached path from the initial population. The two are input into the root cause reasoning model to infer the root cause, and generate breakthrough test cases through targeted mutation. The regular test cases and breakthrough test cases are added to the initial population to form the evolved population.

[0012] The self-healing closed-loop module is configured to execute test cases in the evolved population, capture error information generated during execution and input it into the third LLM iteration for repair, store the repaired test cases with comprehensive fitness exceeding the threshold to the test case library, and output a test case set for the code under test.

[0013] Optionally, the semantic heuristic initialization module is further configured to: perform static analysis on each reference code in the codebase to extract the control flow graph and abstract syntax tree of each reference code, convert the control flow graph and abstract syntax tree into structural feature vectors, and establish the mapping relationship between the structural feature vectors and the corresponding test cases and store them in a vector database to build a test case library.

[0014] Optionally, the semantic heuristic initialization module is further configured to: perform static analysis on the code to be tested to generate a target structural feature vector, and use the target structural feature vector as a query term to retrieve reference code and its test cases with similar structural feature vectors from the test case library.

[0015] Optionally, the semantic potential score includes at least one dimension of the logical penetration score, the data construction rationality score, and the boundary coverage potential score; the logical penetration score is used to evaluate whether the input data of the test case can penetrate the pre-logic of the code under test and reach the core nested logic; the data construction rationality score is used to evaluate the semantic correlation between parameters in the test case and the rationality of the object initialization state; the boundary coverage potential score is used to evaluate whether the input of the test case is within the critical value neighborhood of the code predicate judgment condition or covers a specific semantic extreme value.

[0016] Optionally, the targeted breakthrough evolution module is further configured to: use the extracted logical constraints, the test cases whose execution trajectories obtained from the initial population are closest to the unreached paths, and the object state information of the code under test as diagnostic context input to the root cause reasoning model; the root cause reasoning model, based on the semantic understanding of the code logic and test cases, infers and analyzes the root cause that the test cases cannot meet the logical constraints and thus cannot reach the unreached paths; the root cause includes at least one of the following: incomplete object state initialization, input parameters not falling into the critical value range, or the calling order not meeting the preconditions.

[0017] Optionally, the targeted breakthrough evolution module is further configured to: adjust key variables or execution steps in the test case that hinder path access based on root causes; wherein the adjustment includes at least one of adjusting boundary parameter values, completing function calls, or rearranging the object initialization order.

[0018] Optionally, the self-healing closed-loop module is further configured to store test cases that have passed the repair and whose overall fitness exceeds the threshold, along with their reusable experience, into a test case library; the reusable experience includes the distribution pattern of the input parameters of the test cases, the object construction sequence, and the obstacle-breaking experience for specific logical branches.

[0019] Secondly, embodiments of the present invention provide an LLM-based evolutionary testing method, which is applied to an LLM-based evolutionary testing system as described in any of the first aspects; the evolutionary testing method includes:

[0020] Using RAG technology, reference code and test cases with similar structures to the code under test are retrieved from the test case library. The search results are then integrated with the code under test and its contextual dependency information to form prompt words, which are then input into the first LLM to generate the initial population.

[0021] Each test case in the initial population is executed to obtain path coverage and the overall coverage results are statistically analyzed. Each test case is input into the second LLM to obtain a semantic potential score, and the overall fitness of each test case is calculated based on the path coverage and semantic potential score.

[0022] The system selects test cases from the initial population and performs logical hybridization to generate regular test cases. When the update of the overall coverage result is deemed to be stalled, it locates the unreached path based on the overall coverage result, extracts logical constraints, and obtains the test cases whose execution trajectories are closest to the unreached path from the initial population. The two test cases are then input into the root cause reasoning model to infer the root cause, and breakthrough test cases are generated through directional mutation. The regular test cases and breakthrough test cases are then added to the initial population to form the evolved population.

[0023] Execute test cases in the evolved population, capture error information and input it into the third LLM iteration for repair, store the repaired test cases with comprehensive fitness exceeding the threshold to the test case library, and output a test case set for the code under test.

[0024] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and running on it, wherein the processor executes the computer program to implement the LLM-based evolutionary testing method described in the second aspect.

[0025] Fourthly, it stores a computer program that, when executed by a processor, implements the LLM-based evolutionary testing method described in the second aspect.

[0026] (III) Beneficial Effects

[0027] This application provides an LLM-based evolutionary testing system, method, and apparatus. First, a heuristic population initialization mechanism based on retrieval-enhanced generation is used. This mechanism leverages the characteristics of the tested method to retrieve reference cases with high similarity in functional logic and dependency dimensions from the existing codebase. These reference cases serve as semantic context to drive the large language model to generate an initial population of logically sound test cases, thus establishing a high-quality evolutionary starting point. Next, a dynamic closed-loop guidance stage integrating multi-dimensional scoring and adaptive prompts is implemented. This stage captures the physical path coverage of test cases in real time and combines this with semantic potential scoring of test cases by the large language model to construct a hybrid evaluation model for accurately measuring individual fitness. The prompt template is dynamically adjusted based on the scoring results to forcibly strengthen the subsequent logical modeling. Then, during evolutionary execution, the system uses the large language model to execute semantic-level genetic crossover operators to achieve logical recombination. Specifically, when the overall coverage enters a stagnant period, the system triggers a targeted breakthrough mechanism, injecting uncovered branch constraints and code context into the large language model. This leverages the model's logical reasoning capabilities to generate targeted mutants capable of penetrating deep logic or boundary conditions, guiding the search process out of local optima. Finally, to ensure the stability of the evolutionary closed loop, the system introduces a self-repair mechanism based on multi-turn dialogue and a long-term memory module. This mechanism repairs abnormal stack information generated during the evolution process in real time and stores the generated valid test cases, repair strategies, and scoring feedback in the test case library. This ensures that the system can refer to historical decisions and continuously optimize and iterate. Through the cyclical collaboration of the above four stages, the system ultimately outputs an optimized test case set with high coverage and strong semantic rationality. This solves the technical problems of existing search-based technologies, which are prone to blindness and local optima due to a lack of semantic understanding, and the inability of large language models to autonomously associate coverage indicators and the difficulty of systematically solving the problem of continuous coverage optimization due to shallow fusion of the two. Attached Figure Description

[0028] Figure 1 A schematic diagram of an LLM-based evolutionary testing system provided in an embodiment of this application is shown;

[0029] Figure 2 This paper illustrates a semantic heuristic initialization flowchart provided by an embodiment of this application.

[0030] Figure 3 This paper illustrates a flowchart of a multi-dimensional collaborative evaluation method provided by an embodiment of this application.

[0031] Figure 4 This paper illustrates a general flowchart of a targeted breakthrough evolution provided by an embodiment of this application;

[0032] Figure 5 This paper shows a model structure diagram of a CARIT model provided in an embodiment of this application;

[0033] Figure 6 This paper illustrates a self-healing closed-loop overall flowchart provided by an embodiment of this application;

[0034] Figure 7 A flowchart of an LLM-based evolutionary testing method provided in an embodiment of this application is shown. Detailed Implementation

[0035] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] While current automated test generation technologies have made some progress in both search algorithms and large language models, from the core objective of test coverage optimization, existing technologies still struggle to achieve continuous and stable coverage improvements in complex software systems. In practical applications, whether relying solely on search-based test generation methods or directly using large language models for test generation, the problem of slow or even stagnant coverage growth is prevalent, and a technological system capable of effectively overcoming the constraints of complex paths and continuously guiding test evolution has yet to be established.

[0037] One major drawback of Search-Based Test Generation (SBST) is that its coverage rate easily stagnates. Traditional SBST methods typically use coverage as the fitness function, iteratively evolving test cases through operators such as random mutation and crossover. However, the mutation process of these methods lacks an understanding of program semantics and path constraints. As tests gradually cover shallow logic, the search space shrinks rapidly, and random perturbations struggle to generate valid inputs that satisfy deep branch conditions, causing the evolutionary process to get stuck in local optima. At this point, even with continued investment of computational resources, the improvement in coverage is extremely limited, and the test generation process exhibits a clear plateau characteristic.

[0038] On the other hand, while test generation methods based on large language models offer significant advantages in code understanding, semantic modeling, and test case readability, their core drawback lies in their inability to directly and effectively improve test coverage. Large language models are essentially probabilistic generative models, prioritizing syntactic correctness and semantic rationality over covering specific unreached paths or branch conditions. Without explicit coverage feedback and search constraints, the model struggles to determine "which paths are not yet covered" and "how to modify the input to reach these paths," resulting in generated test cases that, while formally reasonable, contribute little to actual coverage.

[0039] Furthermore, while existing solutions combining SBST with large language models attempt to address the aforementioned shortcomings, most remain at a relatively superficial level of integration. For example, they may only invoke the large model to generate new test seeds when a search fails or coverage stagnates, or use the large model as a separate supplementary generation module. In these solutions, the large language model does not truly participate in the core decision-making process of the evolutionary algorithm, lacking a systematic understanding of historical search states, coverage evolution trajectories, and uncovered targets. Its role is more of an "emergency patch," failing to fundamentally solve the problem of continuous coverage optimization.

[0040] Based on this, this application provides an LLM-based evolutionary testing system, method, and apparatus. First, through a heuristic population initialization mechanism based on retrieval-enhanced generation, reference cases with high similarity in functional logic and dependency dimensions are retrieved from the existing codebase using the characteristics of the method under test. These reference cases are then used as semantic context to drive the large language model to generate an initial population of test cases with a logical foundation, thus establishing a high-quality evolutionary starting point. Next, a dynamic closed-loop guidance stage integrating multi-dimensional scoring and adaptive prompts is entered. By capturing the physical path coverage of test cases in real time and combining it with the semantic potential scoring of test cases by the large language model, a hybrid evaluation model is constructed to accurately measure individual fitness. The prompt template is dynamically adjusted based on the scoring results to forcibly strengthen the subsequent generated logical modeling. Then, during the evolutionary execution process, the system uses the large language model to execute semantic-level genetic crossover operators to achieve logical recombination. Specifically, when the overall coverage is detected to be stagnant, the system triggers a targeted breakthrough mechanism, injecting uncovered branch constraints and code context into the large language model. Utilizing its logical reasoning capabilities, it generates targeted mutants capable of penetrating deep logic or boundary conditions, guiding the search process out of local optima traps. Finally, to ensure the stability of the evolutionary closed loop, the system introduces a self-repair mechanism based on multi-turn dialogue and a long-term memory module to repair abnormal stack information generated during the evolution process in real time, and to store the generated valid test cases, repair strategies and scoring feedback in the test case library. This ensures that the system can refer to historical decisions and continuously optimize and iterate. Through the cyclical collaboration of the above four stages, the system finally outputs an optimized test case set with high coverage and strong semantic rationality.

[0041] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0042] First embodiment:

[0043] Please see Figure 1 , Figure 1 A schematic diagram of an LLM-based evolutionary testing system provided in an embodiment of this application is shown. Figure 1 As shown, the system includes:

[0044] The semantic heuristic initialization module is configured to retrieve reference code and its test cases with similar structure to the code under test from the test case library based on Retrieval-augmented Generation (RAG) technology. The reference code and its test cases, along with the code under test and its contextual dependency information, are integrated into semantic enhancement prompts and then input into the first large language model (LLM) to generate the initial population.

[0045] The multidimensional collaborative evaluation module is configured to execute each test case in the initial population to obtain the path coverage of each test case, generate an overall coverage result based on the path coverage of all test cases, input each test case into the second LLM to obtain the semantic potential score of each test case, and calculate the comprehensive fitness of each test case based on the path coverage and semantic potential score of each test case.

[0046] The targeted breakthrough evolution module is configured to select test cases from the initial population, perform logical hybridization to generate regular test cases, and when it is determined that the update of the overall coverage result has stalled (e.g., the improvement rate is lower than a preset threshold according to the overall coverage results of multiple consecutive generations recorded by the multi-dimensional collaborative evaluation module), it locates the unreached path based on the overall coverage result of the current generation, extracts the logical constraints required to trigger the unreached path, and obtains test cases that are close to the unreached path from the initial population. The logical constraints and the obtained test cases that are close to the unreached path are input into the root cause reasoning model to reason about the unreachable root cause, and generate breakthrough test cases through targeted mutation. The generated regular test cases and breakthrough test cases are added to the initial population to form the evolved population.

[0047] The self-healing closed-loop module is configured to execute test cases in the evolved population, capture error information generated during execution and input it into the third LLM for iterative repair, store the repaired test cases with a comprehensive fitness exceeding the threshold to the test case library, and output a high-quality test set for the code under test.

[0048] It should be understood that the specific functions and structures of each module in the system can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0049] Optionally, this semantic heuristic initialization module builds upon the underlying capabilities of large language models, guiding the model to shift from lexical semantic memorization to deep engineering structure modeling through systematic structural feature induction and sample augmentation. This module is generally divided into three main stages, and its flowchart is shown below. Figure 2 As shown, specifically:

[0050] During the knowledge base construction phase, this module performs in-depth static analysis on the reference code of existing projects, extracting method-level control flow graphs and abstract syntax trees. Using graph embedding algorithms, these topological structures representing the code execution logic are transformed into structural feature vectors in a high-dimensional space. These vectors not only quantify the number of branches and loops but also encode the sequence and nesting level of logical nodes, thus accurately depicting the "logical shape" of the code. Subsequently, this module establishes a mapping relationship between structural feature vectors and corresponding test cases and stores them in a vector database, thereby constructing an associated index library from the underlying topology to high-level testing experience.

[0051] In other words, the semantic heuristic initialization module is further configured to: perform static analysis on each reference code in the codebase to extract the control flow graph and abstract syntax tree of each reference code, convert the control flow graph and abstract syntax tree into structural feature vectors, and establish the mapping relationship between the structural feature vectors and the corresponding test cases and store them in the vector database to build a test case library.

[0052] Furthermore, in the logic-driven semantic retrieval phase, for the code under test, this module uses its structural feature vector as the query term to perform precise retrieval based on structural similarity. By comparing control flow features laterally, the system filters out reference code that is highly similar in terms of branch density, loop depth, and nesting complexity. These reference cases carry path exploration experience when facing similar logical forms and can provide transferable path inspiration for the code under test. Since the retrieved test cases implicitly contain object construction and environment initialization logic, the system reuses these mature initialization patterns, enabling the generated test cases to quickly establish an executable entry state. This significantly improves the execution success rate and entry point coverage of test cases without the need for explicit modeling of complex dependency environments.

[0053] In other words, the semantic heuristic initialization module is further configured to: perform static analysis on the code to be tested to generate a target structural feature vector, and use the target structural feature vector as a query term to retrieve reference code and its test cases with similar structural feature vectors from the test case library.

[0054] Furthermore, in the coverage-guided population generation phase, this module transforms the retrieved structured knowledge into generation instructions for a large language model. First, it constructs semantically enhanced prompts, aggregates the code under test and its contextual dependencies, and injects retrieved reference methods and their test cases as logical templates to ensure the initial population is logically sound. Based on this, the prompts introduce path resolution guidance operators to induce the model to execute thought chain reasoning, performing semantic-level decomposition and deduction of the deep branches of the code under test before generating test cases. Through this "analyze first, construct later" heuristic mechanism, the model can accurately deduce the key input parameters required to penetrate complex logical barriers, thereby generating an initial population that is widely distributed within the logical space and has high coverage potential. This fundamentally solves the technical pain point of traditional search methods, which struggle to reach deep paths due to a lack of semantic guidance.

[0055] It should also be understood that the information contained in this context-dependent information can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0056] For example, this contextual dependency information includes, but is not limited to: the definition of the class to which the code under test belongs (i.e., the class definition), the member variables involved in that class, other methods called internally by the code under test, and related environment configuration information. This information is extracted from the codebase through static analysis and combined with test cases generated from the large model to ensure that the generated test cases can correctly construct objects, initialize states, and satisfy execution dependencies. Specifically, the definition of the class to which the code under test belongs refers to the complete definition of the class, including the class name, parent class, and implemented interfaces; the member variables involved in that class refer to the class's attribute definitions, including variable names, types, and access modifiers; other methods called internally by the code under test refer to the method signatures and brief functionalities of other methods called internally by the code under test; and the environment configuration information refers to the dependent environment and configuration files required for the code under test to run.

[0057] It should also be understood that the specific model and its structure of the first LLM can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0058] For example, the first LLM can be a closed-source commercial model, including but not limited to: GPT-4 series models (such as GPT-4, GPT-4o, GPT-4 Turbo), Claude series models (such as Claude 3.5 Sonnet), Gemini series models, etc. These models perform well in code generation and semantic understanding, and can generate high-quality test cases based on prompts.

[0059] For example, this first LLM can be an open-source or locally deployable code-large language model, including but not limited to: CodeLlama series models (such as CodeLlama-7B, CodeLlama-13B, CodeLlama-34B), DeepSeek-Coder series models, Mistral 7B and Mixtral 8x7B models, Qwen2.5-Coder series models, M2.1-Coding model, etc. These models support private deployment and can run without relying on external APIs, making them suitable for scenarios with high data security requirements.

[0060] Therefore, this semantic heuristic initialization module utilizes RAG technology to retrieve similar business logic as semantic context, guiding the large model to generate initial test cases with real business backgrounds. This module solves the problem of "excessive search space" caused by random initialization in traditional methods by pre-creating seed individuals with deep logical penetration capabilities, allowing the evolutionary process to start from a high-coverage starting point.

[0061] Optionally, this multi-dimensional collaborative evaluation module constructs a hybrid fitness evaluation system that integrates physical coverage and semantic potential scoring. During the evaluation process, this module not only relies on physical feedback such as line coverage or branch coverage generated during real-time execution, but also introduces a large language model as a semantic judge. It performs in-depth scanning and potential assessment of test cases from three dimensions: logical penetration, data construction rationality, and boundary coverage potential. Its overall flowchart is shown below. Figure 3 As shown, specifically:

[0062] In terms of logical penetration, this module inputs the control flow graph of the code under test and the input sequence of the test case into the second LLM. The second LLM then uses static reasoning to evaluate whether the input data can penetrate the pre-guard statements of the code under test and reach the core nested logic. Based on the evaluation results, the second LLM outputs a logical penetration score for the test case.

[0063] It's important to note that this dimension is designed to address the "evaluation plain" problem in the early stages of evolution. In the initial stages of evolution, many test cases have not yet generated actual physical coverage, making it difficult for traditional fitness functions to distinguish between superior and inferior individuals. By assigning higher scores to test cases with penetration potential, the system can effectively identify and retain high-quality genes that, while not yet triggering coverage, are logically close to deeper branches. This prevents them from being prematurely eliminated during the selection process due to a lack of physical coverage, thus ensuring population diversity and evolutionary potential.

[0064] Furthermore, regarding the data construction rationality dimension, the second LLM receives context dependency information of the test cases and the code under test as input. It utilizes its pre-trained domain knowledge to analyze the semantic relationships between parameters in the test cases and detects whether the object initialization state conforms to common business logic. Based on the analysis results, the second LLM outputs a data construction rationality score for the test case.

[0065] It's important to note that this dimension is designed to avoid ineffective searches during the evolutionary process, specifically preventing the generation of numerous test cases that cannot be executed due to data logic conflicts. By improving the rationality score, the system can filter out data combinations that meet business constraints, ensuring the population evolves within a legitimate search space. This significantly reduces the execution crash rate caused by incomplete object states, contradictory parameter semantics, or missing dependencies, thereby improving the overall survival quality and evolutionary efficiency of the population.

[0066] Furthermore, regarding boundary coverage potential, the second LLM receives test cases and predicate judgment conditions from the code under test as input, and identifies whether the input data of the test case is within the critical value neighborhood of the predicate judgment condition, or whether it covers a specific semantic extreme value. Based on the identification results, the second LLM outputs the boundary coverage potential score of the test case. Here, the predicate judgment condition refers to the logical judgment expression in the code under test used to control the program execution path, usually manifested as a Boolean expression in conditional statements (such as if, while, etc.). These expressions determine the branching direction of the program execution flow by comparing variables with constants, and the relationships between variables (such as greater than, less than, etc.).

[0067] It's important to note that this dimension is designed to overcome deep logical barriers, namely complex boundary conditions that conventional search methods struggle to reach precisely. By awarding bonuses to test cases with boundary detection features, the system can drive the population towards the logical edge, giving higher retention priority to individuals with critical value detection capabilities or semantic extreme value coverage capabilities. This guides evolutionary search to penetrate boundary regions that are difficult for conventional methods to cover, achieving precise breakthroughs in complex boundary conditions.

[0068] It should also be noted that, in one embodiment of this application, the logic penetration score, data construction rationality score, and boundary coverage potential score are all completed by the same second LLM through different prompt words; in other embodiments, multiple different second LLMs may also undertake evaluation tasks of different dimensions. Those skilled in the art can configure them according to actual needs, and this invention does not limit this.

[0069] For example, GPT-4 can be used as a second LLM, and different prompt words can be designed to complete the evaluation tasks in three dimensions: prompt word A guides the model to focus on logical penetration assessment, prompt word B guides the model to focus on data construction rationality assessment, and prompt word C guides the model to focus on boundary coverage potential assessment; as another example, CodeLlama-34B can be used as a code-specific model to complete logical penetration assessment, while GPT-4 is used to complete the boundary coverage potential assessment that requires broader domain knowledge.

[0070] The above models are merely illustrative examples. Those skilled in the art can select appropriate large language models according to actual needs. The scope of protection of this invention is not limited to the specific models listed.

[0071] In other words, the semantic potential score includes at least one dimension of logical penetration score, data construction rationality score, and boundary coverage potential score; the logical penetration score is used to evaluate whether the input data of the test case can penetrate the pre-logic of the code under test and reach the core nested logic; the data construction rationality score is used to evaluate the semantic relationship between parameters in the test case and the rationality of the object initialization state; the boundary coverage potential score is used to evaluate whether the input of the test case is within the critical value neighborhood of the code predicate judgment condition or covers a specific semantic extreme value.

[0072] Furthermore, within the comprehensive evaluation system, this module constructs a collaborative evaluation mechanism primarily driven by physical coverage feedback and supplemented by large-scale model semantic scoring. Specifically, when calculating the comprehensive fitness of each test case, the multi-dimensional collaborative evaluation module prioritizes physical coverage data generated during real-time execution to ensure that the evolutionary process has objective metric support. Based on this, it overlays logical penetration scores, data construction rationality scores, and boundary coverage potential scores generated by the second LLM evaluation, thereby enabling refined evaluation of individuals that cannot be effectively distinguished by physical metrics. Here, physical coverage refers to the code coverage metric actually measured through real-time execution of test cases, used to quantify the execution reach of the code under test by the test cases. As an objective feedback basis in evolutionary testing, physical coverage reflects the code elements already reached by the test cases in the real-world operating environment, rather than the potential coverage capability obtained through static analysis or semantic reasoning.

[0073] Regarding the specific calculation method of comprehensive fitness, this invention provides several exemplary implementation schemes. For example, the multidimensional collaborative evaluation module can adopt a weighted summation method, which weights and sums the physical coverage data with the three semantic potential scores. The weight coefficients can be dynamically adjusted according to the evolutionary stage. For example, in the early stage of evolution, the weight of the semantic score can be appropriately increased to solve the "evaluation plain" problem, while in the later stage of evolution, the weight of the physical coverage can be increased to strengthen coverage orientation. Another example is the hierarchical screening method, which first sorts the test cases based on the physical coverage data as the main indicator. For individuals with the same or similar physical coverage, a more refined sort is then performed based on the weighted sum of the three semantic potential scores, thereby achieving a semantic-level distinction between superior and inferior cases while ensuring the priority of objective indicators. For example, a threshold-based method can be used, with physical coverage data as the base value. When a certain semantic potential score exceeds a preset threshold, a corresponding bonus coefficient is added to the base value. This is suitable for scenarios where it is desirable to highlight individuals with specific semantic potential without distinguishing ordinary individuals. Another example is the machine learning fusion method, which pre-collects historical evolutionary data, uses physical coverage data and three semantic potential scores as features, and uses the final success of covering deep paths as a label to train a lightweight fusion model. During the evolution process, the predicted value output by the model is used as the comprehensive fitness, thereby automatically learning the non-linear relationship between physical coverage and semantic features.

[0074] It should be noted that the above calculation method is merely an illustrative example. Those skilled in the art can choose an appropriate fusion strategy based on the actual application scenario. The scope of protection of this invention is not limited to the specific algorithms listed. Regardless of the calculation method used, the core lies in combining physical coverage feedback with semantic potential scoring to achieve a comprehensive measurement of test cases from "execution results" to "logical intent".

[0075] It's also important to note that this dual-track evaluation method aims to leverage the deep understanding of code logic provided by large language models to overcome the inherent blind spots of traditional physical feedback in perceiving individual evolutionary potential. Specifically, it targets test cases that, while not yet providing actual physical coverage, possess the logical potential to penetrate deeper branches. By integrating comprehensive measurements of both "execution results" and "logical intent," this module can accurately identify and retain high-potential genes, effectively addressing the "evaluation plain" problem in early evolution where a lack of physical coverage hinders individual differentiation. This guides the population to continuously penetrate deeper into the code's logical regions, thereby improving overall evolutionary efficiency and final coverage.

[0076] Therefore, this multi-dimensional collaborative evaluation module constructs a hybrid evaluation index that combines physical coverage and semantic potential scoring, breaking the limitations of relying solely on a single-path index. By dynamically adjusting prompts based on real-time execution feedback, the system enables large models to accurately identify and strengthen test logic with high coverage potential, achieving goal-oriented guidance for the evolutionary process.

[0077] Optionally, this targeted breakthrough evolution module is used to address the common problem of code coverage growth stagnation in the later stages of evolutionary testing. When traditional evolutionary algorithms repeatedly iterate but fail to reach the remaining code paths, the system identifies the causes of stagnation, analyzes the conditions of uncovered paths, and generates targeted test cases to effectively break through complex logic branches. Its overall flowchart is as follows: Figure 4 As shown, specifically:

[0078] During the evolutionary iteration process, the system continuously monitors changes in the overall coverage result of the population. When the improvement in the overall coverage result across multiple generations is consistently below a preset threshold (i.e., the update of the overall coverage result stagnates), it is determined that the current search process has entered a local optimum, making it difficult to further improve the coverage rate through conventional logical hybridization or random mutation operations. The improvement in the overall coverage result refers to the increase or growth rate of the current population's overall coverage result relative to the previous generation over multiple generations. At this point, the system automatically switches from a "global random exploration" mode to a "target-driven local breakthrough" mode. In the global random exploration mode, the system primarily maintains population diversity through conventional evolutionary operations such as logical hybridization, broadly exploring potential high-coverage areas within the overall search space. When switching to the target-driven local breakthrough mode, the system focuses on specific code paths that have not yet been reached, initiating a targeted breakthrough evolution module to perform root cause analysis and targeted mutation operations to precisely overcome complex logical barriers.

[0079] Furthermore, during the local breakthrough phase, the targeted breakthrough evolution module first analyzes the overall coverage results of the current population, locating lines of code or conditional branches that have not yet been reached by any test cases as target paths to be broken through. Subsequently, this module uses static analysis and control flow backtracking techniques to reverse-engineer the path conditions required to trigger the target path, transforming the originally complex program condition expressions into structured logical constraint descriptions. These logical constraint descriptions present, in a standardized form, the variable value relationships, object state requirements, and call order dependencies required to trigger the target path. The lines of code or conditional branches that have not yet been reached refer to the specific code locations identified by the system through analyzing the code coverage reports after the execution of all test cases in the current population, locations that have not been covered by any test case execution.

[0080] Thus, through this process, the system not only identifies "which code has not yet been covered," i.e., pinpoints specific unreached paths, but also further analyzes "why existing test cases cannot trigger this code," revealing the gap between existing test cases and the target path's admission conditions through logical constraint descriptions. This analysis provides a structured diagnostic basis for subsequent root cause reasoning model input.

[0081] Furthermore, in order to generate new test cases that can reach uncovered paths, the targeted breakthrough evolution module constructs a diagnostic context containing various key information, specifically including the following three types of information:

[0082] First, the object state information of the code under test, that is, the internal data state of the object under test before the execution of the test cases, including the current values ​​of member variables, the degree of object initialization completion, and the injection status of dependent objects. This information is used to evaluate whether the existing test cases meet the execution prerequisites of the target path at the object construction level;

[0083] Second, the logical constraints that the target path must satisfy, namely, the variable value relationships, object state requirements, and call order dependencies that must be met to trigger unreached paths, extracted through static analysis and control flow backtracking. This information is presented in a structured form, clearly revealing the admission conditions of the target path;

[0084] Third, the test case whose execution trajectory is closest to the target path in the current population, i.e., the existing test case selected from the initial population that is most similar to the target path in terms of control flow. This test case serves as a reference sample, providing the closest possible test pattern to success.

[0085] This targeted breakthrough evolution module integrates the three types of information mentioned above into a diagnostic context, which is then input into the root cause reasoning model. Based on its semantic understanding of the code logic and test cases, the root cause reasoning model compares the differences between the actual execution path of the test case and the logical constraints required to trigger the target path, and infers the root cause that prevents the test case from reaching the target path. Through this diagnostic mechanism, the system can accurately identify the specific factors hindering the path from being reached, providing a clear direction for subsequent targeted mutation operations.

[0086] Therefore, through the above reasoning analysis, the root cause reasoning model can identify the specific root cause that prevents the test case from reaching the target path, and the root cause includes, but is not limited to, the following types:

[0087] Incomplete object state initialization: When the test case is executed, the member variables of the object under test are not correctly initialized, dependent objects are not injected, or the initial state does not meet the execution prerequisites of the target path.

[0088] Input parameters do not fall within the critical value range: The parameter values ​​passed by the test case fail to meet the value range required by the predicate judgment conditions in the target path, such as not reaching the critical threshold, not covering boundary values, or not containing specific semantic extreme values;

[0089] Call order does not meet preconditions: The order of method calls in the test case fails to meet the state dependencies implied by the target path, such as not executing necessary preparation methods before calling the method under test, or modifying the object state at the wrong time.

[0090] Based on the specific root causes identified by the root cause reasoning model, the targeted breakthrough evolution module no longer performs aimless random mutations, but instead generates test adjustment plans with clear objectives. Furthermore, these adjustment plans precisely target the key factors hindering path access, pointing to the variables or execution steps that need modification, providing clear guidance for subsequent targeted mutation operations. For example, if the root cause is "input parameters not falling within the critical value range," the adjustment plan will indicate the specific parameters that need adjustment and their target value range; if the root cause is "incomplete object state initialization," the adjustment plan will indicate the object construction steps that need to be completed or the initialization sequence that needs to be corrected.

[0091] In other words, the targeted breakthrough evolution module is further configured to: use the extracted logical constraints, the test cases whose execution trajectories obtained from the initial population are closest to the unreached paths, and the object state information of the code under test as diagnostic context input to the root cause reasoning model. Based on the semantic understanding of the code logic and test cases, the root cause reasoning model infers and analyzes the root cause that the test cases cannot meet the logical constraints and thus cannot reach the unreached paths. The root cause includes at least one of the following: incomplete object state initialization, input parameters not falling into the critical value range, or the calling order not meeting the preconditions.

[0092] Subsequently, the targeted breakthrough evolution module transforms the root cause analysis conclusions identified by the root cause reasoning model into specific targeted mutation operations, making targeted modifications to key variables or execution steps in the test cases that hinder the path to reach the target. Furthermore, this targeted mutation operation follows the "principle of minimal modification," meaning that while preserving as much of the original correct logic of the test case as possible, it only makes precise adjustments to the local parts that hinder the target path, thereby achieving small but effective changes and significantly increasing the probability that the test case reaches the target code path.

[0093] Specifically, targeted mutation operations include, but are not limited to, the following types:

[0094] Adjust boundary parameter values: For cases where the root cause is "input parameters not falling within the critical value range," modify the parameter values ​​in the test cases to critical or typical values ​​that can satisfy the path constraints. For example, if the target path requires x > 5 and the current parameter value is 3, then adjust x to a value that can trigger the condition, such as 6 or 5.1; if the target path involves boundary value detection, then adjust the parameter to a value within the critical value neighborhood (such as slightly greater than the threshold, slightly less than the threshold, or equal to the boundary value).

[0095] Complete necessary function calls: For cases where the root cause is missing dependencies due to "call order not meeting preconditions" or "incomplete object state initialization," add missing method calls or dependency injection statements to the test cases. For example, complete the object initialization method, pre-preparation method, or instantiation code of dependent services before calling the method under test to ensure that the object under test is in the correct state before execution;

[0096] Reorder object initialization: For cases where the root cause is an incorrect order of calls ("call order does not meet preconditions"), adjust the execution order of existing method calls in the test cases to conform to the state dependencies implicit in the target path. For example, move business methods that were originally called before object property settings to after property settings, or move dependency injection operations to before object usage.

[0097] The breakthrough test cases generated through the aforementioned targeted mutation operations precisely fix local defects that hinder path access while preserving the original correct logical framework, thereby achieving maximum coverage gain with minimal modification cost. These breakthrough test cases will be reintroduced into the population to participate in subsequent evolutionary iterations.

[0098] Ultimately, the targeted breakthrough evolution module reintroduces the generated breakthrough test cases into the current population, forming an evolved population to participate in the next round of evolutionary iterations. During the introduction process, the system assigns these breakthrough test cases a higher retention priority. Specifically, when the multidimensional collaborative evaluation module performs the next round of fitness calculation, breakthrough test cases are given an initial comprehensive fitness bonus, or their probability of being selected is increased during the selection operation, thereby preventing their critical changes from being prematurely destroyed by random perturbations in subsequent evolutionary processes.

[0099] Through the aforementioned mechanism, this application constructs a closed-loop system of deep collaboration between a large language model and an evolutionary algorithm: the root cause reasoning model is responsible for root cause analysis and the generation of targeted mutation strategies, accurately "breaking through barriers" to generate test cases capable of penetrating complex logical barriers; the evolutionary algorithm (evolutionary iteration engine) is responsible for maintaining the overall search space and optimizing the population, maintaining population diversity through conventional logical crossover operations, and retaining superior individuals through selection mechanisms. Both perform their respective functions and cooperate with each other, enabling the system to continuously break through complex logical branches in the later stages of evolution, achieving a stable improvement in code coverage.

[0100] Furthermore, the semantic understanding capability of large language models compensates for the blindness of traditional evolutionary algorithms in local search, while the global search framework of evolutionary algorithms provides a population foundation and an environment for iterative evolution for targeted breakthroughs of large language models. The two complement each other and together constitute an intelligent test generation system with continuous self-optimization capabilities.

[0101] It should also be understood that the specific model and its structure of the root cause reasoning model can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0102] For example, if the root cause reasoning model can be the fourth largest language model, it can adopt a general-purpose large language model based on the Decoder-Only architecture, such as GPT-4, LLaMA-3, or Qwen-72B. These models perform well in complex logical reasoning tasks due to their powerful context understanding and generation capabilities. Alternatively, code-specific models pre-trained on code corpora, such as CodeLlama-34B, DeepSeek-Coder-33B, or Qwen2.5-Coder-32B, can be selected. These models have a deeper understanding of code structure, control flow, and data dependencies, and can more accurately resolve differences in logical constraints and execution paths. Furthermore, Encoder-Decoder architecture models (such as T5 and BART) can also be used for this type of task, and they also perform well in sequence-to-sequence conversion tasks.

[0103] For example, to further improve the performance and interpretability of the root cause reasoning model on root cause reasoning tasks, the existing large language model architecture can be specifically extended to construct an enhanced model for code path reasoning, namely the Code-Aware Root-cause Inference Transformer (CARIT). This model is then used as the root cause reasoning model. Addressing the heterogeneous nature of the diagnostic context and the specific needs of root cause reasoning in this invention, this model introduces dedicated modules such as multi-source input encoding, code structure fusion, path constraint comparison, and multi-task output on top of the standard Transformer architecture. This enables accurate identification of the deep-seated reasons why test cases cannot reach the target path and generates targeted mutation strategies. Its specific model structure is as follows: Figure 5 As shown.

[0104] like Figure 5 As shown, the multi-source input encoding module can perform source-specific encoding on different information sources in the diagnostic context. Specifically, the test case code text is processed by a standard code encoder (such as a Transformer encoder), outputting the hidden state sequence of the code; the constraint encoder (such as a lightweight Transformer encoder or LSTM) parses the logical condition text into an abstract syntax tree or operator-operand sequence, which is then encoded by the lightweight encoder to generate constraint vector representations; object state information is also extracted using a state encoder (such as a Transformer encoder); simultaneously, the control flow graph and abstract syntax tree of the code under test are encoded using graph neural networks (such as GAT or GCN), outputting an embedding matrix of graph nodes. This embedding captures the branching hierarchy, nested loops, and data dependencies of the code. The representations output by each encoder interact in subsequent layers through different fusion mechanisms.

[0105] Furthermore, the code structure fusion layer module seamlessly injects the code topology information extracted by the graph neural network into the textual representation of the test case code. This allows the model to understand the semantics of the code text while simultaneously perceiving the direction of control flow, the hierarchy of branches, and the boundaries of logical blocks, providing rich structural priors for subsequent path constraint comparisons. This code structure fusion layer module receives two types of input: first, the test case code hidden state sequence from the code encoder, which represents the textual semantics of the code at the lexical level; second, the embedding matrix of graph nodes from the graph neural network, which encodes the topological structure information in the control flow graph or abstract syntax tree at the node level, including jump relationships between basic blocks, nesting levels of statements, and data dependency paths. During the fusion process, the model uses the code hidden state sequence as the query and the graph node embedding matrix as the key and value. It calculates the relevance weights between each code position and each graph node through a multi-head cross-attention mechanism, thereby aggregating the structural information of the graph nodes to the corresponding code positions based on relevance weights. After attention calculation, the model adds the original code hidden state to the attention output through residual connections and applies layer normalization, ultimately outputting a structure-enhanced code representation. This representation retains the semantic features of the original code at each lexical position while embedding the topological prior of the code region where the lexical is located. This enables the model to more accurately understand the execution logic of the code in subsequent path constraint comparisons, such as determining the position of a conditional statement in the control flow graph, identifying the depth of nested loops, and locating key variables in data dependencies.

[0106] Furthermore, the core task of this path constraint comparison module is to perform a deep comparison between the structurally enhanced code representation and the logical constraints of the target path, thereby quantifying the differences between the two and providing key differential features for subsequent root cause classification. This module receives two inputs: first, a structurally enhanced code representation from the code structure fusion layer, which has fused code semantics and topological structure information at the lexical level; and second, a constraint vector from the constraint encoder, which encodes the logical conditions required to trigger unreached paths, such as variable value relationships or object state requirements.

[0107] Furthermore, within the path constraint contrast module, the interaction between the structure-enhanced code representation and the constraint vector is first calculated through a contrastive attention sublayer. Specifically, this sublayer uses each lexical position in the structure-enhanced code representation as a focus, calculating its relevance score to the constraint vector. This identifies the parts of the code sequence most closely related to the target path constraint and simultaneously captures potential mismatches between these parts and the constraints. Through this process, the model generates a contrast vector reflecting local differences, highlighting key positions in the code that hinder path reach and their deviation from the constraints. The contrastive attention sublayer generates the relevance score by calculating the dot product between each code position and the constraint vector, or through learnable attention weights.

[0108] Meanwhile, the path constraint comparison module also calculates the similarity between the overall semantics of the structure-enhanced code representation and the constraint vector, serving as a global difference metric. This similarity assesses the closeness between the test case execution path and the target path constraints at a holistic level, providing a macro-level perspective for subsequent root cause analysis.

[0109] Ultimately, the path constraint comparison module outputs a comparison vector and a similarity score, both of which together form part of the input to the subsequent multimodal fusion decoder. The comparison vector provides refined local difference features for root cause classification, while the similarity score serves as a global guidance signal, enabling the model to integrate local and global information in subsequent inference and more accurately pinpoint the specific reasons why test cases cannot reach the target path.

[0110] Furthermore, this multimodal fusion decoder module deeply fuses multi-source information from preceding modules and generates root cause reasoning autoregressively. This module receives three core inputs: first, a structure-enhanced code representation from the code structure fusion layer, which integrates code semantics and topological information; second, an object state information representation from the state encoder, which encodes the internal data state of the object under test before executing the test case; and third, a comparison vector from the path constraint comparison module, which quantifies the local differences between the test case execution path and the target path constraints. In addition, the similarity score calculated by the comparison module is also incorporated into the decoding process as a global guidance signal.

[0111] Inside the decoder, the three representations are first concatenated along the sequence dimension to form an initial hidden state sequence containing multi-source information. This sequence is then fed into a multi-layer Transformer decoder, each layer containing a masked multi-head self-attention mechanism, a cross-attention mechanism, and a feedforward neural network. Causal masking ensures the decoding process follows an autoregressive principle, meaning that the generation of each position can only depend on information from previous positions, thus simulating the step-by-step reasoning process of human experts. Within each layer, masked self-attention makes the model focus on previously generated historical information; cross-attention allows the decoder to interact with the original inputs (such as code hidden states and graph node embeddings) from the encoder, continuously reviewing and calibrating the reasoning basis; and the feedforward network performs a non-linear transformation on the fused representation to extract higher-order features.

[0112] Through multi-layered, progressive processing, the decoder fully interacts and integrates execution path information from the structure-enhanced code representation, initialization details from the object state information, difference features from the comparison vector, and global guidance signals from the similarity score. Finally, the last layer outputs a hidden state sequence carrying all the information required for root cause analysis. This sequence contains a deep understanding of "why existing test cases cannot reach the target path" and also includes the knowledge needed to generate mutation strategies, providing a rich feature foundation for subsequent multi-task output layers.

[0113] Furthermore, this multi-task output module simultaneously performs two core tasks—root cause classification and mutation strategy generation—based on the hidden state sequence generated by the preceding decoder. This multi-task output module receives the hidden state sequence from the last layer of the multimodal fusion decoder as input. This sequence has been fully integrated during the decoding process with structurally enhanced code representation, object state information, contrast vectors, and similarity scores, carrying all the semantic and structural features required for root cause analysis.

[0114] In the root cause classification task, the hidden state sequence output by the decoder is first pooled. When focusing on the generation guided by the start symbol, the hidden state corresponding to the first special label (such as the start symbol) in the sequence is usually taken; or when global inference is required, the hidden states at all positions are averaged to obtain a fixed-dimensional global representation vector. This vector is then input into a classification head composed of a multilayer perceptron, processed through several linear transformations and nonlinear activation functions, and finally output as a multi-label probability vector through a sigmoid activation function. Each dimension in this vector corresponds to a root cause type, including incomplete object state initialization, input parameters not falling into the critical value range, and call order not meeting preconditions, etc., and its value represents the confidence level of the existence of the root cause type. This multi-label design allows the model to simultaneously identify multiple coexisting hindering factors, providing a comprehensive diagnostic basis for subsequent mutation operations.

[0115] In the mutation policy generation task, the model shares the hidden state sequence of the decoder and uses language modeling for autoregressive generation. Specifically, the hidden state sequence output by the decoder is input into the language modeling head, which is typically a linear layer with a softmax function, used to predict the probability distribution of the next word. The generation process uses the inference steps already generated by the decoder as context, generating mutation instructions in natural language word by word, such as "adjust parameter x to 6 and complete the initialize method before calling it". This instruction details the specific modifications that need to be made to the test case, including the target value of the parameter adjustment, the function call to be completed, and the initialization order to be rearranged.

[0116] Ultimately, the multi-label probability vector output by the root cause classification head provides downstream modules with a clear indication of the root cause type, while the natural language instructions output by the mutation strategy generation head can be parsed into specific code modification operations by the targeted mutation module. Together, they form a closed-loop chain from root cause diagnosis to mutation execution, enabling CARIT not only to accurately identify the problem but also to directly guide the generation of solutions.

[0117] Therefore, through the above extended structure, the root cause reasoning model can not only utilize the general knowledge of large language models, but also combine code structure priors and specialized comparison mechanisms to accurately identify the deep-seated reasons that prevent test cases from reaching the target path, and generate targeted mutation schemes, significantly improving the success rate of targeted breakthroughs.

[0118] It should be noted that the above extended structure is only an exemplary implementation. Those skilled in the art can add, delete, or adjust the modules according to actual needs. Its core lies in enhancing the root cause reasoning ability of the model through code structure awareness and multimodal fusion.

[0119] Therefore, this targeted breakthrough evolution module addresses the coverage stagnation problem commonly encountered in the later stages of evolution by extracting deep branch constraints from uncovered paths and leveraging the logical reasoning capabilities of large models for targeted mutation. By precisely penetrating complex boundary conditions and logical barriers, it forcibly triggers deep code paths that are difficult to reach using traditional search methods.

[0120] Optionally, the purpose of this self-healing closed-loop module is to ensure the robustness of the evolutionary process and to continuously optimize the generation strategy through knowledge accumulation. Its overall flowchart is as follows: Figure 6 As shown, specifically:

[0121] For invalid individuals generated during evolutionary mutations or large model generation, a targeted error correction link based on execution feedback is established. When a test case triggers an exception during the compilation phase or runtime, the system's configured execution monitoring component intercepts the error information in real time and refines it into compiler diagnostic logs, exception stack traces, and related context dependencies (such as class definitions and member variables of the code under test). This module utilizes a third-party LLM to deeply analyze these multi-dimensional feedbacks, locating syntax defects, logical conflicts, or missing environment configurations that cause execution interruption, and constructing prompts with clear repair instructions accordingly. After receiving error feedback, the third-party LLM performs targeted fine-tuning of the test case code, such as correcting illegal API calls, refactoring object initialization sequences, or completing missing dependency injections. This repair process is not a single trigger but adopts a multi-round iterative closed-loop model. This module allocates a preset attempt quota for each individual to be repaired. Within the quota, the third-party LLM continuously evolves by referring to the failure results of previous repair attempts until the test case successfully passes execution verification or reaches the iteration limit. This mechanism can effectively save intermediate individuals that carry excellent logical characteristics but have grammatical flaws, significantly improving the overall survival of the population.

[0122] Furthermore, after obtaining test cases that can be compiled correctly, the self-healing closed-loop module establishes a mechanism for screening and accumulating high-quality test cases based on fitness thresholds. This mechanism directly connects to the scoring results of the multi-dimensional collaborative evaluation module, rather than storing all test cases in their entirety. The module sets strict admission rules: only when a test case passes compilation and runtime verification and its overall fitness score exceeds a preset high-score threshold is it considered a logic template with long-term storage value. These high-scoring test cases are structured and stored in the test case library, which includes not only the test case source code but also the distribution patterns of its corresponding input parameters, object construction sequences, and obstacle-breaking experience for specific logical branches. Through this screening and storage method, the test case library evolves into a high-density index of testing experience.

[0123] The input parameter distribution pattern refers to the parameter value characteristics obtained from statistical analysis of high-scoring test cases, including the data type distribution of each parameter, common value ranges, boundary value distribution, and collaborative change patterns among multiple parameters. This pattern is obtained through cluster analysis or frequent pattern mining of test case parameter values ​​and is used to guide the reasonable selection of parameter values ​​during subsequent test generation, avoiding execution failures caused by blindly selecting random values.

[0124] The object construction sequence refers to the order of method calls executed by high-scoring test cases when creating the object under test and setting its state. Specifically, it includes the order of object instantiation statements, member variable assignment operations, dependency injection methods, and preparatory functions. This sequence is obtained through static analysis of the test case code and extraction of the method call chain from the abstract syntax tree. It can serve as a logical template for object initialization, guiding new test cases to correctly construct object states.

[0125] Breakthrough experience for specific logical branches refers to the strategic knowledge employed when test cases successfully reach complex logical branches that were previously difficult to cover. This includes the path constraints of the target branch, the types of root causes identified during the diagnostic phase (such as incomplete object state, parameters not reaching critical values, or incorrect call order), the targeted mutation operations used (such as parameter adjustment, function completion, and order reordering), and the applicable conditions of the strategy. Breakthrough experience is recorded during the process of successfully generating breakthrough test cases from the targeted breakthrough evolution module and is an important carrier for the system's transformation from "trial and error" to "experience."

[0126] In other words, the self-healing closed-loop module is further configured to store test cases that have passed the repair and whose overall fitness exceeds the threshold, along with their reusable experience, into a test case library; the reusable experience includes the distribution pattern of the input parameters of the test cases, the object construction sequence, and the obstacle-breaking experience for specific logical branches.

[0127] Furthermore, high-scoring test cases stored in the test case library play a guiding role in subsequent evolutionary cycles. When the semantic heuristic initialization module encounters new code to be tested or is in a subsequent iteration cycle, the retrieval operator extracts the optimal test template from the test case library based on the similarity of the code structure. These templates serve as logical references for the generation of the first major language model, providing successful code patterns for subsequent test generation. Through this feedback loop based on long-term memory, the system not only avoids reproducing previously corrected syntax errors but also learns from the evolutionary paths of high-scoring test cases, guiding the evolutionary search process towards higher coverage potential. This closed-loop design ensures that the test generation system has the ability to continuously learn and self-optimize, achieving high efficiency and robustness in test case generation under complex engineering environments. The retrieval operator, within the semantic heuristic initialization module, is the functional component responsible for finding high-scoring test cases from the test case library that are similar in structure to the current code to be tested.

[0128] Therefore, this self-healing closed-loop module introduces a multi-round self-healing and long-term memory mechanism based on execution feedback. By capturing runtime anomalies to drive iterative repair of the large model, it maximizes the retention of "ill" use cases with high-value coverage potential and accumulates repair strategies. This closed-loop ecosystem ensures that test evolution continues to evolve through knowledge accumulation, ultimately maximizing coverage gains.

[0129] In summary, by leveraging the aforementioned technical solutions, this application effectively addresses the problems of traditional SBST methods, such as the lack of semantic guidance and blind search direction during the mutation process, and the lack of path-level guidance in existing large-model-assisted methods, by constructing a collaborative mechanism of "perception-diagnosis-targeted breakthrough." This application utilizes a static analysis engine to explicitly parse uncovered paths, extract corresponding logical constraints, and drive the large language model to perform differentiated path deduction, transforming the originally difficult-to-quantify search stagnation state into a structured logical breakthrough target. Compared to Codemosa-like solutions that only call the large language model to generate new test cases during the evolutionary stagnation stage, this application, through a targeted mutation strategy oriented towards the target path, can significantly improve the coverage of deeply nested logical branches.

[0130] Furthermore, regarding the quality of test case evolution and search efficiency, this application introduces a collaborative evaluation mechanism that integrates physical coverage feedback and multidimensional semantic potential, overcoming the problems of traditional methods that rely on a single evaluation metric and struggle to distinguish potentially high-quality individuals. The system utilizes a large language model to assist in the evaluation of the semantic structure and execution potential of test individuals, enabling the effective identification and retention of individuals that have not yet generated immediate coverage increments but possess subsequent evolutionary value. This guides the population to continuously evolve towards more accessible search areas, improving the stability and efficiency of the overall search process.

[0131] Furthermore, regarding system robustness and knowledge accumulation capabilities, this application overcomes the limitations of existing technologies where test cases are directly eliminated due to grammatical or structural defects, and search experience is difficult to reuse across generations, by constructing a self-healing closed-loop mechanism and a long-term memory module. The system utilizes a multi-round repair strategy driven by a large language model to perform semantic-level repair on test cases with potential coverage value but initially unexecutable, enabling them to re-participate in the evolutionary process. Simultaneously, based on the knowledge accumulation mechanism of Retrieval Enhanced Generation (RAG), high-value test cases are abstracted into reusable logical templates, providing a reference for subsequent test initialization and mutation processes, thereby enhancing the system's continuous adaptability and stable evolutionary capabilities in complex engineering scenarios.

[0132] Furthermore, this application introduces a Retrieval Enhanced Generation (RAG) mechanism to provide key contextual information such as coverage, path constraints, and historical evolutionary data to the large language model in a structured manner, enabling it to deeply participate in the mutation, generation, and selection processes of the evolutionary algorithm. By constructing a closed-loop collaborative mechanism between search feedback and semantic generation, this application transforms the large language model from a passive generation tool into an evolutionary operator with coverage awareness, effectively overcoming the coverage stagnation problem of SBST and compensating for the inherent shortcomings of purely large model-driven methods in terms of coverage, thus achieving continuous and targeted improvement in test coverage.

[0133] It should be understood that the above-described LLM-based evolutionary testing system is merely exemplary, and those skilled in the art can make various modifications based on the system described above, and such modified solutions also fall within the protection scope of this application.

[0134] Second embodiment:

[0135] Please see Figure 7 , Figure 7 A flowchart of an LLM-based evolutionary testing method provided in an embodiment of this application is shown. The method is applied to an LLM-based evolutionary testing system as described in the first embodiment; as... Figure 7 As shown, this evolutionary testing method includes:

[0136] Step S710: Use RAG technology to retrieve reference code and test cases with similar structure to the code under test from the test case library. After integrating the retrieval results with the code under test and its context dependency information into prompt words, input them into the first LLM to generate the initial population.

[0137] Step S720: Execute each test case in the initial population to obtain path coverage and statistically analyze the overall coverage results, and input each test case into the second LLM to obtain a semantic potential score, and calculate the comprehensive fitness of each test case based on the path coverage and semantic potential score;

[0138] Step S730: Select test cases from the initial population to generate regular test cases through logical hybridization; when the update of the overall coverage result is determined to be stagnant, locate the unreached path based on the overall coverage result, extract logical constraints, and obtain the test cases whose execution trajectories are closest to the unreached path from the initial population. Input the two into the root cause reasoning model to reason about the root cause, and generate breakthrough test cases through directional mutation. After adding the regular test cases and breakthrough test cases to the initial population, the evolved population is formed.

[0139] Step S740: Execute test cases in the evolved population, capture error information and input it into the third LLM iteration for repair, store the repaired test cases with comprehensive fitness exceeding the threshold to the test case library, and output the test case set for the code under test.

[0140] It should be understood that the specific details of each step in this method can be found in the relevant content of the first embodiment, and will not be described in detail here.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0143] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0144] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0145] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0146] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. An evolutionary testing system based on LLM, characterized in that, include: The semantic heuristic initialization module is configured to use RAG technology to retrieve reference code and test cases with similar structures to the code under test from the test case library, integrate the retrieval results with the code under test and its context dependency information into prompt words, and then input them into the first LLM to generate the initial population. The multidimensional collaborative evaluation module is configured to execute each test case in the initial population to obtain path coverage and statistically analyze the overall coverage results, input each test case into the second LLM to obtain a semantic potential score, and calculate the comprehensive fitness of each test case based on the path coverage and the semantic potential score. The targeted breakthrough evolution module is configured to select test cases from the initial population, perform logical hybridization to generate regular test cases, and when it is determined that the update of the overall coverage result has stalled, locate the unreached path based on the overall coverage result, extract logical constraints, and obtain the test cases whose execution trajectories are closest to the unreached path from the initial population. The two are input into the root cause reasoning model to infer the root cause, and generate breakthrough test cases through targeted mutation. The regular test cases and the breakthrough test cases are added to the initial population to form the evolved population. The self-healing closed-loop module is configured to execute test cases in the evolved population, capture error information generated during execution and input it into the third LLM iteration for repair, store the repaired test cases with comprehensive fitness exceeding the threshold into the test case library, and output a test case set for the code under test.

2. The LLM-based evolutionary testing system according to claim 1, characterized in that, The semantic heuristic initialization module is further configured to: perform static analysis on each reference code in the code library to extract the control flow graph and abstract syntax tree of each reference code, convert the control flow graph and the abstract syntax tree into structural feature vectors, and establish a mapping relationship between the structural feature vectors and the corresponding test cases and store them in a vector database to construct the test case library.

3. The LLM-based evolutionary testing system according to claim 2, characterized in that, The semantic heuristic initialization module is further configured to: perform static analysis on the code to be tested to generate a target structural feature vector, and use the target structural feature vector as a query term to retrieve reference code and its test cases with similar structural feature vectors in the test case library.

4. The LLM-based evolutionary testing system according to claim 1, characterized in that, The semantic potential score includes at least one dimension of logical penetration score, data construction rationality score, and boundary coverage potential score; the logical penetration score is used to evaluate whether the input data of the test case can penetrate the pre-logic of the code under test and reach the core nested logic. The data construction rationality score is used to evaluate the semantic relationship between parameters and the rationality of object initialization state in the test cases; The boundary coverage potential score is used to evaluate whether the input of the test case is within the critical value neighborhood of the code predicate judgment condition or covers a specific semantic extreme value.

5. The LLM-based evolutionary testing system according to claim 1, characterized in that, The targeted breakthrough evolution module is further configured to: input the extracted logical constraints, the test case whose execution trajectory is closest to the unreached path obtained from the initial population, and the object state information of the code under test as diagnostic context into the root cause reasoning model; the root cause reasoning model, based on the semantic understanding of the code logic and the test case, infers and analyzes the root cause that the test case cannot meet the logical constraints and thus cannot reach the unreached path; the root cause includes at least one of incomplete object state initialization, input parameters not falling into the critical value range, or the calling order not meeting the preconditions.

6. The LLM-based evolutionary testing system according to claim 5, characterized in that, The targeted breakthrough evolution module is further configured to: adjust the key variables or execution steps that hinder path access in the test case according to the root cause; wherein, the adjustment includes at least one of adjusting boundary parameter values, completing function calls, or rearranging the object initialization order.

7. The LLM-based evolutionary testing system according to claim 1, characterized in that, The self-healing closed-loop module is further configured to store test cases that have passed repair and whose overall fitness exceeds the threshold, along with their reusable experience, into the test case library; the reusable experience includes the distribution pattern of the input parameters of the test cases, the object construction sequence, and the obstacle-breaking experience for specific logical branches.

8. An evolutionary testing method based on LLM, characterized in that, The evolutionary testing method is applied to an LLM-based evolutionary testing system as described in any one of claims 1 to 7; The evolutionary testing method includes: Using RAG technology, reference code and test cases with similar structures to the code under test are retrieved from the test case library. The retrieval results are integrated with the code under test and its contextual dependency information to form prompt words, which are then input into the first LLM to generate the initial population. Each test case in the initial population is executed to obtain path coverage and the overall coverage result is statistically analyzed. Each test case is input into the second LLM to obtain a semantic potential score, and the overall fitness of each test case is calculated based on the path coverage and the semantic potential score. Test cases are selected from the initial population and generated through logical cross-pollination. When it is determined that the update of the overall coverage result has stalled, unreached paths are located based on the overall coverage result, logical constraints are extracted, and test cases whose execution trajectories are closest to the unreached paths are obtained from the initial population. The two are input into the root cause reasoning model to infer the root cause, and breakthrough test cases are generated through targeted mutation. The regular test cases and the breakthrough test cases are added to the initial population to form the evolved population. Execute the test cases in the evolved population, capture error information and input it into the third LLM iteration for repair, store the repaired test cases with comprehensive fitness exceeding the threshold into the test case library, and output the test case set for the code under test.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on it. When the processor executes the computer program, it implements the LLM-based evolutionary testing method of claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the LLM-based evolutionary testing method as described in claim 8.

Images