Test case automatic generation and verification method and device based on large language model

Abstract

The present application relates to the technical field of computer, especially to a test case automatic generation and verification method and device based on a large language model, the method comprising: parsing an on-board software requirement document to generate a structured requirement text; receiving an on-board software source code, parsing the on-board software source code to extract function information of each function therein and generating a function semantic abstract for each function, thereby forming an abstract function data set integrating the function information and the corresponding function semantic abstract; establishing a semantic correspondence between the requirements and the code based on the structured requirement text and the abstract function data set to generate a requirement-code matching list; generating a test case set satisfying the requirement constraints based on the structured requirement text, the abstract function data set and the requirement-code matching list; executing the test cases in the test case set and generating a test report, thereby solving the problems of low efficiency and strong artificial dependence of the traditional scheme.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for automatically generating and verifying test cases based on a large language model. Background Technology

[0002] With the rapid development of the modern aviation industry, airborne software, as the core carrier for flight control, navigation communication, and real-time processing of high-precision sensors (such as gyroscopes and inertial navigation systems), is experiencing exponential growth in functionality, scale, and complexity. On the one hand, airborne software must meet stringent real-time requirements, such as high sampling rates and low data latency for gyroscope data, to ensure accurate transmission of flight status monitoring and control commands. On the other hand, it must strictly adhere to software engineering standards such as GJB 438C-2021, which details the types, structures, formats, and contents of airborne software documentation, while also imposing extremely high requirements on hardware and software interface compatibility, environmental adaptability, security, and reliability.

[0003] Traditional requirement-driven testing solutions, as the current mainstream practice, mainly rely on manual or semi-automatic methods to complete requirement analysis, test design, and test case execution. While this approach is feasible to some extent when dealing with structured requirements under the GJB 438C-2021 standard, its parsing efficiency drops significantly when faced with unstructured, complex requirements described in natural language, and it is prone to missing edge cases and abnormal scenarios. Furthermore, the core test logic still needs to be written manually, resulting in low automation. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method and apparatus for automatic generation and verification of test cases based on a large language model, which solves the technical problem of low automation in the prior art.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, embodiments of the present invention provide a method for automatically generating and verifying test cases based on a large language model, comprising: receiving an onboard software requirement document; parsing the onboard software requirement document to generate a structured requirement text; receiving onboard software source code; parsing the onboard software source code to extract function information of each function and generating a functional semantic summary for each function, thereby forming a summary function data set integrating function information and corresponding functional semantic summaries; establishing a semantic correspondence between requirements and code based on the structured requirement text and the summary function data set to generate a requirement-code matching list; generating a test case set that satisfies the requirement constraints based on the structured requirement text, the summary function data set, and the requirement-code matching list; executing the test cases in the test case set and generating a test report.

[0009] Optionally, the airborne software requirements document is parsed to generate structured requirements text, including: identifying the physical structure of the airborne software requirements document based on a document parsing library to extract text paragraphs and table elements containing requirements information; analyzing the heading styles and chapter numbers in the text paragraphs to determine the requirements hierarchy, and constructing a requirements tree structure based on the determined hierarchy; attaching the extracted text paragraph content to the corresponding nodes in the requirements tree structure, and converting the content of the extracted table elements into a structured data format and associating it with the corresponding nodes in the requirements tree structure; and generating and outputting structured requirements text based on the requirements tree structure and the content attached to its nodes.

[0010] Optionally, the structured requirements text includes at least one of the functional requirements, performance requirements, interface requirements, and constraints parsed from the airborne software requirements document.

[0011] Optionally, the airborne software source code is parsed to extract function information for each function and generate a functional semantic summary for each function, including: generating an abstract syntax tree of the airborne software source code based on the compilation toolchain; traversing the abstract syntax tree to extract function information for each function; the function information includes the function body source code and function interface feature information, and the function interface feature information includes at least one of the following: function name, position information in the source file, parameter list, and return value type; inputting the function body source code of each function into the first large language model to generate a functional semantic summary describing its functional semantics.

[0012] Optionally, structured requirement text is used to represent the requirement tree structure, which includes multiple requirement nodes. Each requirement node contains a title, associated body content, and related structured table data extracted from the onboard software requirement document. A semantic correspondence between requirements and code is established based on the structured requirement text and a dataset of functions with summaries. This includes: converting each requirement node in the structured requirement text into requirement description text; preprocessing the function information in the dataset of functions with summaries to form function description text, including: truncating function body source code exceeding a threshold length and / or adjusting the expression of the function summary to match its semantic granularity with the requirement description text; calculating the semantic similarity between each requirement description text and each function description text; and matching one or more functions for each requirement node based on the semantic similarity, generating a requirement-code matching list containing matching scores.

[0013] Optionally, generating a test case set that meets the requirements constraints includes: based on symbolic execution technology, analyzing the code paths of functions in the requirements-code matching list and solving for constraints to generate a first test case covering multiple paths; according to a preset prompt word template, creating a model input for each function in the requirements-code matching list based on its source code and the corresponding matching requirement description text, and inputting the model into a second large language model to obtain the natural language logical analysis text and structured test data output by the model, and generating a second test case based on the natural language logical analysis text and structured test data; and merging and deduplicating the first and second test cases to form the final test case set.

[0014] Optionally, test cases in the test case set are executed, and a test report is generated, including: running the test case set in the test execution environment and monitoring its execution process; obtaining execution result data and code coverage data generated during test execution; generating a test report based on the execution result data, code coverage data, and the correlation information in the requirement-code matching list; wherein, the test report integrates the mapping relationship between requirements, test cases, tested code, and execution results to achieve full-link traceability of the test process.

[0015] Secondly, embodiments of the present invention provide a test case automatic generation and verification system based on a large language model, comprising:

[0016] The requirements parsing module is used to receive airborne software requirements documents, parse the airborne software requirements documents, and generate structured requirements text.

[0017] The code parsing module is used to receive the airborne software source code, parse the airborne software source code to extract the function information of each function, and generate a functional semantic summary for each function, thereby forming a set of summary function data that integrates function information and corresponding functional semantic summaries;

[0018] The requirement-code mapping module is used to establish a semantic correspondence between requirements and codes based on structured requirement text and a dataset with summary functions, in order to generate a requirement-code matching list;

[0019] The test case generation module is used to generate a set of test cases that meet the requirements constraints based on structured requirement text, a dataset with summary functions, and a requirement-code matching list.

[0020] The test case execution module is used to execute test cases in the test case set and generate test reports.

[0021] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the automatic generation and verification method for test cases based on a large language model as described in any of the first aspects.

[0022] Fourthly, embodiments of the present invention provide a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to execute the automatic generation and verification method for test cases based on a large language model as described in any of the first aspects.

[0023] (III) Beneficial Effects

[0024] The beneficial effects of this invention are:

[0025] This invention proposes a method and apparatus for automatic generation and verification of test cases based on a large language model. By constructing a complete closed loop of "requirement analysis - code analysis - requirement and code mapping - test case generation and execution feedback", it solves the problems of low efficiency and strong reliance on manual intervention in traditional solutions. Attached Figure Description

[0026] Figure 1 The flowchart illustrates a method for automatic generation and verification of test cases based on a large language model, as provided in an embodiment of this application.

[0027] Figure 2 This paper presents an overall flowchart of an airborne software requirements-driven automated testing framework based on a large language model, according to an embodiment of this application.

[0028] Figure 3 This paper illustrates a general flowchart of a requirement analysis method provided in an embodiment of this application.

[0029] Figure 4 This paper illustrates a general flowchart of code parsing provided by an embodiment of this application;

[0030] Figure 5 This paper illustrates an overall flowchart of a requirements-code mapping provided by an embodiment of this application;

[0031] Figure 6 This paper illustrates a general flowchart for generating test cases according to an embodiment of the present application.

[0032] Figure 7 A schematic diagram of a test case generation prompt word template provided in an embodiment of this application is shown;

[0033] Figure 8 This document illustrates an overall flowchart of test case execution provided in an embodiment of this application. Detailed Implementation

[0034] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] As software scales, the manual maintenance cost of requirement traceability matrices rises sharply. When the code exceeds 100,000 lines, the update cycle often extends to several weeks, significantly reducing the traceability between requirements, code, and test cases. In recent years, requirement understanding and test case generation schemes based on large language models have gradually become a research hotspot. This method leverages the natural language understanding and text generation capabilities of large language models, offering certain advantages in requirement parsing and test case generation. However, its application in the airborne software domain still faces several key limitations: insufficient test case coverage, making it difficult to effectively cover edge scenarios and complex logic branches; limited test prediction capabilities, unable to accurately determine whether functionality and performance meet standard requirements; difficulties in model engineering implementation, insufficient performance of small and medium-sized open-source models, and long and costly construction cycles for high-quality domain test datasets.

[0036] In addition, existing solutions still suffer from insufficient accuracy and poor interpretability in the semantic mapping of requirements and code. Especially in multi-step reasoning and naming ambiguity scenarios, the matching accuracy and link consistency are low, making it difficult to form a stable and reliable automated testing loop.

[0037] Based on this, this invention proposes a method and apparatus for automatic generation and verification of test cases based on a large language model. By constructing a complete closed loop of "requirement analysis—code analysis—requirement-code mapping—test case generation and execution feedback," it integrates symbolic execution technology and large language model capabilities to automatically identify and extract functional, performance, interface, and constraint information from airborne software requirement documents. By generating high-coverage, structured, and interpretable test cases, this invention significantly improves the automation level and traceability of airborne software testing, solving problems such as low efficiency, strong reliance on manual intervention, and incomplete coverage in traditional solutions. Simultaneously, this invention enhances the accuracy of requirement-code mapping, improves the completeness and execution efficiency of the test chain, and provides high-reliability support for software testing of critical equipment such as airborne gyroscopes and inertial navigation systems.

[0038] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0039] First embodiment:

[0040] Please see Figure 1 , Figure 1 A flowchart illustrating an automatic test case generation and verification method based on a large language model, provided in an embodiment of this application, is shown. It should be understood that this method can be executed by an electronic device, and the specific device of the electronic device can be configured according to actual needs; this embodiment is not limited thereto. For example, the electronic device can be a computer, a server, etc. Specifically, the method includes:

[0041] Step S110: Receive the airborne software requirements document, parse the airborne software requirements document, and generate a structured requirements text. The structured requirements text includes at least one of the following: functional requirements, performance requirements, interface requirements, and constraints parsed from the airborne software requirements document. Furthermore, the constraints refer to mandatory restrictions and regulations extracted from the requirements document, excluding specific functions, performance indicators, and interface definitions, concerning the software's implementation method, operating environment, compliance with standards, and development process.

[0042] Optionally, the physical structure of the airborne software requirements document is identified based on a document parsing library to extract text paragraphs and table elements containing requirements information; the heading styles and chapter numbers in the text paragraphs are analyzed to determine the requirements hierarchy, and a requirements tree structure is constructed based on the determined hierarchy; the extracted text paragraph content is attached to the corresponding nodes of the requirements tree structure, and the content of the extracted table elements is converted into a structured data format and associated with the corresponding nodes of the requirements tree structure; based on the requirements tree structure and the content attached to its nodes, a structured requirements text is generated and output.

[0043] To facilitate understanding of step S110, the following will be combined with... Figure 2 and Figure 3 To describe it.

[0044] Specifically, this requirements parsing module automates the structured parsing of unstructured requirements information in airborne software requirements documents conforming to the GJB 438C-2021 standard, providing accurate and traceable requirements input for subsequent testing. Its overall flowchart is as follows: Figure 3 As shown. The input to this requirements analysis module is an airborne software requirements document conforming to a standard format. For example, a gyroscope software requirements document, which contains information such as functions, performance, interfaces, and constraints described in natural language, and is presented in formats including multi-level headings, regular paragraphs, tables, and other unstructured or semi-structured content.

[0045] And, such as Figure 2 and Figure 3 As shown, this requirement parsing module identifies the physical structure of a document based on parsing methods from a document parsing library (such as python-docx), automatically filtering out non-core content such as tables of contents, headers, and footers, thereby accurately extracting all text paragraphs and table elements containing requirement information such as functionality, performance, interfaces, and constraints. Next, the module intelligently analyzes the heading styles and predefined chapter numbering rules in the extracted paragraphs, and uses regular expressions for matching to automatically identify and establish a complete hierarchical relationship of requirements from the system level, subsystem level, to specific function point level, thus constructing a hierarchical, top-down requirement tree structure. Subsequently, the module directly attaches the extracted ordinary paragraph text content to the corresponding heading node in the requirement tree, while automatically converting the content of table elements (including complex structures such as nested tables) into machine-readable structured data formats (such as JSON) and associating them with the corresponding nodes in the tree. This process ensures comprehensive and lossless extraction of requirement information even in complex scenarios such as multi-page paragraphs.

[0046] To further ensure output quality, the requirements parsing module automatically verifies and logically supplements the initially constructed structure to correct formatting issues such as missing title numbers or ambiguous paragraph affiliations. Ultimately, the module outputs machine-readable structured requirements text (e.g., JSON format) and a visual requirements tree structure. The structured requirements text is stored in a machine-readable format for easy access by subsequent modules. It also generates multi-level readable files to support manual review and requirements verification, and the requirements tree structure clearly presents the hierarchical relationships of requirements, providing visual support for requirements traceability and change impact analysis.

[0047] Step S120: Receive the airborne software source code, parse the airborne software source code to extract the function information of each function, and generate a functional semantic summary for each function, thereby forming a set of summary function data that integrates function information and corresponding functional semantic summaries.

[0048] Optionally, an abstract syntax tree of the airborne software source code is generated based on the compilation toolchain; the abstract syntax tree is traversed to extract the function information of each function; the function information includes the function body source code and function interface feature information, and the function interface feature information includes at least one of the following: function name, position information in the source file, parameter list, and return value type; the function body source code of each function is input into the first large language model to generate a functional semantic summary describing its functional semantics.

[0049] To facilitate understanding of step S120, the following will be combined with... Figure 2 and Figure 4 To describe it.

[0050] Specifically, this code parsing module automatically extracts source code information and generates functional summaries for the source code of airborne software (e.g., C / C++ source code), providing precise code-side input for the requirements-code mapping and test case generation modules. Its overall flowchart is as follows: Figure 4 As shown. The input to this code parsing module is a source code file that conforms to the airborne software coding standards. The file content includes complete function definitions, type declarations, and parameter logic, and requires that the parsed compilation parameters be consistent with the actual airborne software operating environment to ensure the accuracy and reproducibility of the parsing results.

[0051] And, such as Figure 2 and Figure 4As shown, compilation parameters are configured to match the actual operating environment of the onboard software, and an Abstract Syntax Tree (AST) is generated based on the Clang toolchain to ensure parsing accuracy. Subsequently, the AST is recursively traversed to comprehensively extract all types of functions defined within it, including ordinary functions, class member functions, constructors, and destructors. For each function, the code parsing module automatically and structurally records its core elements, such as function name, function body source code text, start and end line numbers in the source file, parameter list, and return type, thus constructing complete function information. This information is organized and stored in a structured data format. This data structure uses a nested dictionary and list approach to ensure clear information hierarchy and efficient access. The function body source code text consists of all the code strings implementing the function's specific logic, covering all statements, control flow, and comments, serving as the basic raw material for semantic understanding. The start and end line numbers in the source file indicate the precise physical location of the function in the code file (start and end lines), providing key coordinates for accurate tracing and location of test results. The parameter list is an ordered set of input parameters declared by the function, where each parameter includes its data type and name, clearly defining the function's input interface. The return value type is the data type specification of the function's output result, defining its output contract.

[0052] After extracting function information, the code parsing module inputs the complete source code text of each function into the first large language model for semantic understanding and analysis. This model generates a concise and accurate natural language description as a functional semantic summary of the function, thus transforming complex program logic into a natural language description. Furthermore, the generated functional summary accurately summarizes the core functionality implemented by the function. For example, the initialization logic of the gyroscope zero-bias self-calibration algorithm and the parsing and verification of communication data frames in the inertial navigation system provide direct evidence for subsequent semantic layer matching.

[0053] It should be understood that the specific model of the first major language model can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0054] For example, this first major language model can take a piece of function source code text as input and generate an accurate and concise natural language summary to summarize the function's functionality, logic, or algorithmic intent. This could be a GPT-4 series model or a Codex model, etc.

[0055] Finally, the code parsing module integrates all the aforementioned function information items for each function with their corresponding functional semantic summaries, organizing them in a structured data format (such as nested dictionaries and lists) to form a function data set with summaries. This set provides input with both accurate code structure and clear functional semantics for subsequent semantic matching and test generation.

[0056] Step S130: Establish a semantic correspondence between requirements and code based on structured requirement text and dataset with summary functions to generate a requirement-code matching list.

[0057] Optionally, structured requirement text is used to represent the requirement tree structure, which includes multiple requirement nodes. Each requirement node contains a title, associated body content, and related structured table data extracted from the onboard software requirement document. A semantic correspondence between requirements and code is established based on the structured requirement text and a dataset of functions with summaries. This includes: converting each requirement node in the structured requirement text into requirement description text; preprocessing the function information in the dataset of functions with summaries to form function description text, including: truncating function body source code exceeding a threshold length and / or adjusting the expression of the function summary to match its semantic granularity with the requirement description text; calculating the semantic similarity between each requirement description text and each function description text; and matching one or more functions for each requirement node based on the semantic similarity, generating a requirement-code matching list containing matching scores.

[0058] To facilitate understanding of step S130, the following will be combined with... Figure 2 and Figure 5 To describe it.

[0059] Specifically, this requirement-code mapping module achieves precise semantic matching between structured requirement texts and code functions, constructs a traceable link between requirements and code, and provides a clear requirement-code correspondence for the subsequent test case generation module. Its overall flowchart is as follows: Figure 5 As shown, the input to this requirement-code mapping module includes structured requirement text from the requirement parsing module and a set of function data with summaries from the code parsing module. The structured requirement text is organized in a hierarchical format, containing requirement titles, body content, and hierarchical information, which can fully express the logical structure and semantic meaning of the requirement; the set of function data with summaries contains function names, source code text, code start and end line numbers, and a function function summary generated by the large language model, which can comprehensively reflect the implementation content and semantic features of the code.

[0060] And, such as Figure 2 and Figure 5As shown, firstly, for each requirement node in the structured requirement information, its title and associated body content are semantically integrated to form a semantically complete natural language requirement description text, ensuring the integrity and consistency of the requirement's semantic expression. Simultaneously, text preprocessing is performed on the function information in the dataset with summarized functions. This preprocessing may include: reasonably truncating the source code of function bodies exceeding a preset threshold in length, and adjusting the expression format of the function function summary to ensure that the semantic granularity of the processed function text matches the requirement description text. The specific value of the preset threshold can be set according to actual needs, and this embodiment is not limited to this.

[0061] Subsequently, each integrated requirement description text and each preprocessed function text are paired as input pairs and fed into the third large language model. Through specific prompt word engineering, this large language model performs deep semantic comparison and understanding, calculates the semantic similarity between each pair of requirement and function texts, and outputs a quantified matching score between 0 and 1. A higher matching score indicates a higher semantic correlation between the function and the requirement.

[0062] It should be understood that the specific model of the third language model can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0063] For example, this third-largest language model can receive a pair of natural language texts from the demand side and the code side respectively, and calculate a matching score that quantifies their semantic relevance. To achieve this function, those skilled in the art can employ different technical approaches. For instance, the model could be an encoder model specifically fine-tuned for contrastive learning or text matching tasks (such as Sentence-BERT, SimCSE, etc.), whose output is designed as a normalized similarity score. Alternatively, powerful general-purpose generative large language models (such as GPT-4, GLM series) can be directly used, with carefully designed few-sample prompts guiding them to follow strict scoring rules and output formats, thereby outputting a structured matching score and a brief explanation.

[0064] Next, the structured results output by the third language model are analyzed, and the precise start and end line numbers of the source code are appended to each matched function entry. All function matches related to the same requirement are sorted according to their matching scores, and then filtered based on a preset semantic similarity threshold to remove entries with low relevance, thus obtaining a preliminary matching relationship with high confidence and ensuring the accuracy and reliability of the output results. The specific value of this preset semantic similarity threshold can be set according to actual needs, and this embodiment is not limited to this.

[0065] Finally, by integrating all the above processing results, a structured requirement-code matching list is generated. This list clearly records the information of the highly relevant function corresponding to each requirement node. The entries include at least: the requirement title and body, the name of the matched function, the start and end positions of the function in the source code, the matching score, and a functional summary of the function, thus fully and traceably showing the semantic relationship between the requirement and the implementation code.

[0066] Step S140: Based on the structured requirement text, the dataset with summary functions, and the requirement-code matching list, generate a test case set that meets the requirement constraints.

[0067] Optionally, based on symbolic execution technology, for functions in the requirement-code matching list, their code paths are analyzed and constraints are solved to generate first test cases covering multiple paths; according to preset prompt word templates, for each function in the requirement-code matching list, a model input is created based on its source code and the corresponding matching requirement description text, and the model is input into the second large language model to obtain the natural language logical analysis text and structured test data output by the model, as well as the second test cases generated based on the natural language logical analysis text and structured test data; the first test cases and the second test cases are merged and deduplicated to form the final test case set.

[0068] To facilitate understanding of step S140, the following will be combined with... Figure 2 and Figure 6 To describe it.

[0069] Specifically, the test case generation module, as the core execution layer of the test framework of this application, aims to integrate symbolic execution technology with the semantic generation capabilities of large language models. Based on structured requirements, the source code to be tested, and the mapping relationship between requirements and code, it automatically generates requirement-constrained test cases with high coverage, structure, and interpretability. Its overall flowchart is as follows: Figure 6 As shown, this test case generation module is used to verify the correctness and compliance of the core logic of the airborne software, providing a high-quality test platform that can be directly run for the test execution phase.

[0070] Furthermore, the input to the test case generation module includes three key types of information to ensure a close correlation between test cases and requirements and code. Specifically: First, the requirement-side input is a structured requirement text from the requirement parsing module, which includes the requirement title, core constraints, and hierarchical information, accurately reflecting the software's functional and performance requirements. Second, the code-side input is a list of functions with summaries from the code parsing module, including function names, complete source code text, start and end line numbers, and functional summaries, used to locate the logic to be tested. Third, the mapping relationship input is a matching list generated by the requirement-code mapping module, used to indicate the high similarity functions and their matching scores corresponding to each requirement fragment, thereby ensuring that the direction of test case generation is consistent with the requirement objectives.

[0071] Furthermore, the core processing of the test case generation module adopts a logical structure of dual-branch parallel processing and unified assembly, achieving dual guarantees of code path coverage and requirement constraint coverage.

[0072] The first branch generates test cases based on symbolic execution technology, focusing on the completeness of the code execution path. Specifically, this branch inputs the source code of the function under test into the symbolic execution tool. This tool automatically traverses the abstract syntax tree of the code through static analysis and symbolic execution, dynamically identifying conditional decision nodes during simulated execution and extracting and accumulating the logical constraints of each branch to form a set of path constraint conditions. Subsequently, the test case generation module calls the constraint solver to solve the satisfiability of this set of path constraints, thereby generating specific input data that can trigger execution on a specific path. This generates input data that can cover multiple paths and conditions, including boundary values, outliers, and extreme values. Furthermore, the generated input data and expected results are organized into structured test cases according to airborne software testing standards. The content includes test case descriptions, the name of the script under test, function names, initial values ​​of variables, and expected values ​​of variables, ensuring that the test cases are directly executable and conform to standardized format requirements.

[0073] The second branch generates requirement-constrained test cases based on the semantic reasoning capabilities of the large language model, focusing on the accuracy and interpretability of the requirement semantics. Specifically, this test case generation module first constructs model input based on a pre-defined, customized prompt word template. This template explicitly defines the test expert role, task objectives, and strict output format constraints. Then, the source code of the objective function and its corresponding requirement text are filled into this template to form complete prompt content, which is then input into the second large language model. The prompt word template is as follows: Figure 7As shown, the second major language model performs semantic reasoning on the input content and outputs in two steps according to the template requirements: first, it outputs the Chinese formal logic analysis process for the test scenario, clarifying the variable processing logic and expected output in the function; second, it outputs structured test case data, describing variable names, input values, and expected values ​​in machine-readable formats such as JSON. Finally, the test case generation module automatically verifies and corrects the format of the model's output to ensure that the data format is consistent and the values ​​are clearly defined, so that the generated test cases can be directly called by the test execution tool.

[0074] It should be understood that the specific model of the second language model can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0075] For example, this second major language model needs to possess the following key capabilities: 1. Deep code semantic understanding capability, able to parse function logic and align it with the requirement description; 2. Complex instruction following and structured output capability, able to strictly adhere to the constraints in the prompt word template regarding step-by-step output (logic analysis first, then structured data) and fixed formats (such as a specified JSON schema); 3. Requirement-based reasoning capability, able to deduce reasonable test input values ​​and expected output results based on the requirement text. Furthermore, those skilled in the art can choose large-scale generative models trained with a hybrid code and natural language approach (such as GPT-4, GLM-4, DeepSeek-Coder, etc.) and stimulate these capabilities through carefully designed prompt word engineering (Few-Shot or Zero-Shot).

[0076] Furthermore, after the test cases for the two branches mentioned above are generated, the test case generation module performs a unified test case assembly and integration process. First, it extracts the metadata contained in the symbolic execution branch, such as the name of the script under test, function name, and variable types, and associates it one-to-one with the requirement constraint test cases generated by the second language model, ensuring that different test cases under the same function are grouped into a unified set. Then, it embeds the Chinese analysis process generated by the second language model as annotations into the corresponding test case headers, forming interpretable test cases so that testers can understand the test case design logic and trace the results. Finally, the test case generation module deduplicates and sorts the integrated test case set, organizing it according to requirement priority, function call order, and test case type, generating a structured, complete, and interpretable requirement constraint test case set. These requirement constraints include functional constraints, performance constraints, and interface constraints.

[0077] Furthermore, the test case generation module outputs a set of requirement-constrained test cases, which can be stored as text files or a structured database. The output includes directly executable test cases and associated information. These executable test cases cover normal, edge, and abnormal scenarios for the core functions of the onboard software, enabling verification of the stability and reliability of algorithms and logic under different conditions. Each test case includes a corresponding requirement identifier, matching function line number, test case generation source, and thought process comments, thus establishing a full-link traceability relationship between requirements, code, and test cases. In addition, the test case generation module provides pre-analysis data on coverage during the output phase, statistically analyzing the number of functions and branches covered by each test case, providing fundamental support for coverage evaluation by the test execution module.

[0078] Step S150: Execute the test cases in the test case set and generate a test report.

[0079] Optionally, test case sets are run in the test execution environment and their execution process is monitored; execution result data and code coverage data generated during test execution are obtained; a test report is generated based on the execution result data, code coverage data, and the associated information in the requirement-code matching list; wherein, the test report integrates the mapping relationship between requirements, test cases, tested code, and execution results to achieve full-link traceability of the test process.

[0080] To facilitate understanding of step S150, the following will be combined with... Figure 2 and Figure 8 To describe it.

[0081] Specifically, the test case execution module is the core closing stage of the airborne software requirements-driven automated testing framework based on a large language model. It primarily receives requirement-constrained test case sets, automates the testing process, and generates test reports with traceable and quantifiable metrics. This verifies the correctness, functional consistency, and compliance of the core logic of the airborne software, providing objective quantitative evidence for software reliability assessment. This module assumes closed-loop verification responsibilities within the framework, achieving closed-loop management of the entire testing process through standardized input, automated execution, and intelligent report generation. Its overall flowchart is shown below. Figure 8 As shown.

[0082] And, such as Figure 2 and Figure 8As shown, the input to the test case execution module is a set of requirement-constrained test cases from the test case generation module. This test case set is structured and conforms to the execution standards of airborne software testing tools. It includes key information fields such as test case usage description, the name of the script under test, the name of the function under test, initial values ​​of variables, and expected values ​​of variables, enabling direct parsing and execution by the test execution environment. Each test case is bound to its corresponding requirement node information, the source code location of the matching function, and the source of the test case generation, ensuring that the test results can be traced back to specific requirements and code logic, achieving end-to-end consistency from requirements to verification. The test case set covers key functional scenarios of the airborne software, including normal function verification, boundary condition testing, and abnormal input simulation. It undergoes format and syntax validation to ensure the completeness of the input and the effectiveness of execution.

[0083] In its core processing, the test case execution module employs an architecture based on Python server management and automated subprocess execution, controlling the entire process according to the workflow of pre-process cleanup, test case updates, test execution, report generation, and result return. First, before test startup, the module initializes the server, launches a Python control terminal with multi-process creation, monitoring, and termination capabilities, and establishes a standardized communication interface to receive test input and return test results. Upon receiving the test case set, the module automatically performs input validation, checking field integrity and variable type matching to ensure the consistency and security of the executed data.

[0084] During the pre-execution environment cleanup phase, the test case execution module automatically detects and terminates any remaining test and compilation processes in the system, releasing occupied system resources and preventing historical processes from interfering with the new test. It also deletes old reports, logs, and cache files generated by the previous test to ensure the uniqueness and accuracy of the current test output.

[0085] During the execution phase, the module imports the validated test case sets in batches into the automated testing tool adapted for airborne software, automatically associating it with the tool's compilation and runtime environment to ensure complete consistency with the actual airborne system's runtime configuration. During test execution, the system runs test cases one by one according to requirement priority and function call order. Each execution sets the runtime parameters based on the initial variable values ​​defined in the test case, executes the target function, and compares the actual output with the expected output to determine whether the test case execution result is pass or fail. If program crashes, timeouts, or memory exceptions occur during execution, the test case execution module automatically captures the exception stack information and variable states, binding them to the corresponding requirement nodes and functions to form a complete fault tracing chain.

[0086] Upon completion of testing, the test case execution module automatically generates a standardized test report. The report includes quantified coverage metrics and detailed execution logs. Coverage metrics encompass multi-dimensional quantitative data such as statement coverage, branch coverage, MC / DC coverage, function entry point coverage, and function call coverage, and are displayed at the function level, detailing the coverage of different code units. The detailed logs include the execution status, execution duration, associated requirement name, matching function information, and exception details for each test case, ensuring high traceability of the results.

[0087] After the report is generated, the test case execution module automatically packages and compresses all result files into a lightweight test report package, which is then returned to the user through the server interface. This test report includes a main report document, detailed log files, coverage data files, and a requirement traceability table. The main report presents the overall test results and coverage distribution in a visual format. The log files fully record the test execution process and result comparisons. The coverage data files store raw statistical data in a standard structured format, which can be directly parsed by external analysis tools. The traceability table clearly establishes the mapping relationship between requirement nodes, test cases, execution results, and code functions, meeting the specifications for end-to-end traceability in airborne software testing.

[0088] Therefore, leveraging the aforementioned technical solutions, this application utilizes a structured parsing mechanism to automatically identify and hierarchically model unstructured requirement documents under relevant standards, automatically extracting paragraph and table content and generating structured files with clear structure that can be read and processed by machines. Compared to manual methods, parsing efficiency is improved by over 80%, achieving high-precision and comprehensive requirement extraction, ensuring that the results meet software standard requirements.

[0089] Furthermore, regarding the coverage and interpretability of test case generation, this application innovatively integrates symbolic execution and large language model technologies. It generates high-coverage basic test cases through path analysis and constraint solving, and combines model semantic reasoning to generate requirement constraint test cases and Chinese explanations. This mechanism achieves 97.5% coverage of core function statements and 90.3% coverage of branches for test cases, forming interpretable and traceable test results, thus solving the problems of insufficient coverage and "black box" nature in traditional automated testing.

[0090] Furthermore, regarding requirement-code mapping and engineering adaptability, this application employs a large language model semantic matching mechanism to achieve precise association between requirements and code, improving matching accuracy by over 60%. An automated test execution module enables full-process control and coverage analysis, generating reports that include statement, branch, MC / DC coverage, and a "requirement-use case-code" traceability chain. This framework does not rely on large-scale training data, reducing engineering deployment costs by approximately 50%, and can be directly applied to high-reliability testing scenarios such as airborne software.

[0091] It should be understood that the above-described method for automatically generating and verifying test cases based on a large language model is merely exemplary. Those skilled in the art can make various modifications based on the above method, and the modified solutions also fall within the protection scope of this application.

[0092] Second embodiment:

[0093] This invention also provides a test case automatic generation and verification system based on a large language model, the system comprising:

[0094] The requirements parsing module is used to receive airborne software requirements documents, parse the airborne software requirements documents, and generate structured requirements text.

[0095] The code parsing module is used to receive the airborne software source code, parse the airborne software source code to extract the function information of each function, and generate a functional semantic summary for each function, thereby forming a set of summary function data that integrates function information and corresponding functional semantic summaries;

[0096] The requirement-code mapping module is used to establish a semantic correspondence between requirements and codes based on structured requirement text and a dataset with summary functions, in order to generate a requirement-code matching list;

[0097] The test case generation module is used to generate a set of test cases that meet the requirements constraints based on structured requirement text, a dataset with summary functions, and a requirement-code matching list.

[0098] The test case execution module is used to execute test cases in the test case set and generate test reports.

[0099] It should be noted that the specific execution methods of each module in this system can be found in the relevant descriptions in the first embodiment, and will not be repeated here.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0102] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0103] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0105] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for automatic generation and verification of test cases based on a large language model, characterized in that, include: Receive the airborne software requirements document, parse the airborne software requirements document, and generate structured requirements text; The system receives airborne software source code, parses the airborne software source code to extract function information of each function, and generates a functional semantic summary for each function, thereby forming a summary function data set that integrates the function information and the corresponding functional semantic summary. Based on the structured requirement text and the dataset with summary functions, a semantic correspondence between requirements and code is established to generate a requirement-code matching list; Based on the structured requirement text, the dataset with summary functions, and the requirement-code matching list, a test case set that meets the requirement constraints is generated. Execute the test cases in the test case set and generate a test report; Specifically, the airborne software source code is parsed to extract function information for each function, and a functional semantic summary is generated for each function, including: An abstract syntax tree is generated based on the compilation toolchain to produce the source code of the airborne software. Traverse the abstract syntax tree and extract the function information of each function; the function information includes the function body source code and function interface feature information, and the function interface feature information includes at least one of the following: function name, position information in the source file, parameter list, and return value type; Input the source code of the function body of each function into the first large language model to generate a functional semantic summary describing its functional semantics.

2. The method for automatic generation and verification of test cases based on a large language model according to claim 1, characterized in that, The airborne software requirements document is parsed to generate structured requirements text, including: The physical structure of the airborne software requirements document is identified based on the document parsing library in order to extract text paragraphs and table elements containing requirements information. Analyze the heading styles and chapter numbers in the text paragraphs to determine the requirement hierarchy, and construct a requirement tree structure based on the determined hierarchy; The extracted text paragraphs are attached to the corresponding nodes of the requirement tree structure, and the extracted table elements are converted into structured data format and then associated with the corresponding nodes of the requirement tree structure. Based on the aforementioned demand tree structure and the content attached to its nodes, the structured demand text is generated and output.

3. The method for automatic generation and verification of test cases based on a large language model according to claim 2, characterized in that, The structured requirements text includes at least one of the functional requirements, performance requirements, interface requirements, and constraints parsed from the airborne software requirements document.

4. The method for automatic generation and verification of test cases based on a large language model according to claim 1, characterized in that, The structured requirement text is used to represent the requirement tree structure, which includes multiple requirement nodes. Each requirement node contains a title, associated body content, and related structured table data extracted from the airborne software requirement document. The process of establishing a semantic correspondence between requirements and code based on the structured requirement text and the dataset with summary functions includes: For each requirement node in the structured requirement text, each requirement node is converted into requirement description text; The function information in the dataset with summary functions is preprocessed to form function description text. The preprocessing includes: truncating the source code of function bodies that exceed a threshold in length, and / or adjusting the expression of the function function summary to match its semantic granularity with the requirement description text. Calculate the semantic similarity between each of the aforementioned requirement description texts and each of the aforementioned function description texts; Based on the semantic similarity, one or more functions are matched for each requirement node, and a requirement-code matching list containing a matching score is generated.

5. The method for automatic generation and verification of test cases based on a large language model according to claim 1, characterized in that, The generation of a test case set that satisfies the requirement constraints includes: Based on symbolic execution technology, for the functions in the requirement-code matching list, their code paths are analyzed and constraints are solved to generate first test cases covering multiple paths; According to the preset prompt word template, for each function in the requirement-code matching list, a model input is created based on its source code and the corresponding matching requirement description text, and the model is input into the second large language model to obtain the natural language logic analysis text and structured test data output by the model, and a second test case is generated based on the natural language logic analysis text and structured test data. The first test case and the second test case are merged and deduplicated to form the final test case set.

6. The method for automatic generation and verification of test cases based on a large language model according to claim 1, characterized in that, The execution of test cases in the test case set and the generation of a test report include: Run the test case set in the test execution environment and monitor its execution process; Obtain execution result data and code coverage data generated during test execution; Based on the execution result data, code coverage data, and the associated information in the requirement-code matching list, a test report is generated; wherein, the test report integrates the mapping relationship between requirements, test cases, tested code, and execution results to achieve full-link traceability of the testing process.

7. A test case automatic generation and verification system based on a large language model, characterized in that, include: The requirement parsing module is used to receive the airborne software requirement document, parse the airborne software requirement document, and generate structured requirement text; The code parsing module is used to receive the airborne software source code, parse the airborne software source code to extract the function information of each function, and generate a functional semantic summary for each function, thereby forming a summary function data set that integrates the function information and the corresponding functional semantic summary. The requirement-code mapping module is used to establish a semantic correspondence between requirements and codes based on the structured requirement text and the dataset with summary functions, so as to generate a requirement-code matching list; The test case generation module is used to generate a set of test cases that meet the requirement constraints based on the structured requirement text, the dataset with summary functions, and the requirement-code matching list. The test case execution module is used to execute the test cases in the test case set and generate a test report; The code parsing module is specifically used for: generating an abstract syntax tree of the airborne software source code based on the compilation toolchain; traversing the abstract syntax tree and extracting function information of each function; the function information includes function body source code and function interface feature information, and the function interface feature information includes at least one of function name, position information in the source file, parameter list and return value type; inputting the function body source code of each function into the first large language model to generate a functional semantic summary describing its functional semantics.

8. An electronic device, characterized in that, The system includes a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute a method for automatically generating and verifying test cases based on a large language model as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions cause the processor to perform the automatic generation and verification method for test cases based on a large language model as described in any one of claims 1-6.

Images