Intelligent generation and closed-loop optimization method for aviation airborne software test case

Abstract

The invention discloses an aviation airborne software test case intelligent generation and closed-loop optimization method, which comprises the following steps of: knowledge graph construction: analyzing a DO-178C standard document and a related field document, extracting entities and relationships defined in the DO-178C standard document and the related field document, and constructing a field knowledge graph fused with DO-178C standard knowledge; initial test case generation: based on the domain knowledge graph, combining a static analysis result of the source code of the tested airborne software, and utilizing a large language model to drive and generate an initial test case set; and closed-loop iterative optimization: executing the test case, evaluating whether the structural coverage rate reaches the standard or not, automatically identifying uncovered codes when the structural coverage rate does not reach the standard, generating a supplementary test case for iterative optimization, and outputting a final test case set until a coverage rate target corresponding to the software security level is met. According to the method, the test quality and efficiency of the aviation airborne software can be improved.

Description

Technical Field

[0001] This invention relates to the intersection of aviation software verification, software testing, and artificial intelligence, and in particular to a method for intelligent generation and closed-loop optimization of test cases for airborne software. Background Technology

[0002] Airborne software testing for DO-178C standard compliance is a core component of ensuring flight safety and airworthiness certification, and it is also the most resource-intensive and demanding critical path during development. Researching how to efficiently and automatically construct high-coverage test cases can significantly reduce development costs, ensure software reliability, accelerate the certification process, and enhance the overall security of the system.

[0003] Early test case building methods mostly relied on manual writing or automated scripts based on simple rules. Although they could generate effective test cases for clear requirements, they were difficult to handle complex logic and combinations of conditions, making it difficult to achieve structural coverage. Moreover, due to their heavy reliance on the experience of test engineers, they were inefficient. In addition, existing automation solutions focused on the single stage of test case generation and lacked closed-loop integration with requirements tracing, coverage analysis, syntax verification, and resource scheduling, resulting in a fragmented testing process and limited overall efficiency improvement. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide an intelligent generation and closed-loop optimization method for airborne software test cases that combines automation and intelligence, improves the testing quality and efficiency of airborne software, covers the entire process requirements of DO-178C, and integrates testing with the testing of DO-178C.

[0005] Technical solution: The intelligent generation and closed-loop optimization method for airborne software test cases of this invention includes the following steps:

[0006] (1) Knowledge graph construction: Analyze the DO-178C standard document and related domain documents, extract the entities and relationships defined therein, and construct a domain knowledge graph that integrates the knowledge of the DO-178C standard;

[0007] (2) Initial test case generation: Based on the domain knowledge graph and combined with the static analysis results of the source code of the airborne software under test, an initial test case set is generated using a large language model. The generation process includes performing syntax verification on the generated test cases, and if the verification fails, the error information is fed back to the large language model to trigger automatic repair until the verification is passed.

[0008] (3) Closed-loop iterative optimization: (a) Execute the test case set and generate a structural coverage report using the coverage analysis tool; (b) Automatically evaluate whether the structural coverage report meets the coverage target defined in the domain knowledge graph and corresponding to the security level of the software under test; (c) If the coverage target is not met, automatically identify the code structure that has not been covered by the test, take the uncovered code structure information as the new test target, return to step (2), drive the large language model to generate supplementary test cases, and repeat steps (a) to (c); (d) If the coverage target is met, output the final test case set.

[0009] Further, step (1) includes: using a document parsing tool to extract and structurally transform the text of the DO-178C standard document;

[0010] Natural language processing technology is used to perform entity recognition and relation extraction on structured text, and triples containing concepts such as requirements, test cases, coverage criteria, traceability data and their relationships are constructed.

[0011] By using entity linking and disambiguation techniques, triples are stored in a graph database to form the domain knowledge graph.

[0012] Furthermore, step (2), which utilizes a large language model to drive the generation of test cases, includes:

[0013] Based on the domain knowledge graph and static analysis results, normal range test case prompts for verifying the expected behavior of the software and robustness test case prompts for verifying the behavior of the software under abnormal input and boundary conditions are generated.

[0014] The normal range test case prompts and robustness test case prompts are input into the large language model to generate initial test cases covering normal functions and exception handling.

[0015] Furthermore, the design of the normal range test cases and the robustness test cases conforms to the DO-178C standard.

[0016] Furthermore, in the closed-loop iterative optimization of step (3), after identifying the code structure that has not been covered by the test, the uncovered code structure is automatically and backward traced back to the low-level requirements it implements; the low-level requirements are combined with the uncovered code structure information to form a test target with requirement traceability for generating supplementary test cases.

[0017] Furthermore, the structural coverage target is statement coverage, branch coverage, or modified condition / decision coverage.

[0018] Furthermore, the security level of the software under test is the highest level, and the target structure coverage is modified condition / decision coverage.

[0019] An intelligent generation and closed-loop optimization system for airborne software test cases includes: a knowledge graph construction module; a test case generation module; and a closed-loop optimization module.

[0020] Furthermore, the test case generation module uses the domain knowledge graph and static analysis results to generate normal range test case prompts for verifying the expected behavior of the software and robustness test case prompts for verifying the behavior of the software under abnormal input and boundary conditions, respectively, and inputs the prompts into a large language model.

[0021] Furthermore, the closed-loop optimization module will identify uncovered code structures and automatically trace back to their corresponding lower-level requirements, combining the requirements and code structure information into new test targets, and feeding them back to the test case generation module.

[0022] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. It improves the automation and intelligence level of test case construction, shortens the test case design cycle, and effectively improves the completeness and coverage depth of testing; 2. It overcomes the defects of existing tools with single function and broken process, avoids a lot of manual rework, and realizes continuous self-optimization of the testing process; 3. It ensures the compliance of testing activities; 4. It reduces the risk of certification failure due to insufficient test evidence or lack of traceability. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0024] Figure 2 The knowledge graph constructed for this invention;

[0025] Figure 3 This is a schematic diagram illustrating the process of converting PDF files using the open-source framework Mineru in this invention.

[0026] Figure 4 A schematic diagram illustrating the process of constructing a knowledge graph for the field of this invention;

[0027] Figure 5 This is a flowchart illustrating the process of automatically generating test cases based on LLM according to the present invention.

[0028] Figure 6 This is a schematic diagram of the process for verifying the results of this invention. Detailed Implementation

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the intelligent generation and closed-loop optimization method for airborne software test cases includes the following steps:

[0031] The first step is knowledge graph construction:

[0032] Analyze the DO-178C standard document and related domain documents, extract the entities and relationships defined therein, and construct a domain knowledge graph that integrates DO-178C standard knowledge;

[0033] Step 2, Initial Test Case Generation:

[0034] Based on the domain knowledge graph and combined with the static analysis results of the source code of the airborne software under test, an initial test case set is generated using a large language model LLM. The system extracts complete information on specific software requirements from the knowledge graph, including their functional descriptions, interface definitions, data types, value ranges, and performance constraints. Combined with the logical structure obtained from the static analysis of the source code, the system generates two types of structured prompt words.

[0035] Normal range test case prompts: Based on the functional description of the requirements and the effective input range, they aim to verify the expected behavior of the software;

[0036] Robustness test case prompts: Based on the input constraints of the requirements, such as the value range boundary, invalid values, etc., intelligently generate test cases to verify the behavior of the software under abnormal input and conditions;

[0037] Based on these two types of prompts, LLM generates initial test components that cover normal functions and exception handling. These components not only contain test input data and expected output results, but also executable test program scripts, which together constitute the initial form of software verification test cases and programs that conform to the DO-178C definition.

[0038] The generated test cases are not used directly, but are first checked for syntax validity. The system uses an integrated compiler and static analysis tools to perform syntax and static semantic checks on the generated test code. If a syntax error is detected, the system will send the error information to the LLM, triggering it to automatically repair and refactor, generating corrected test cases, until the syntax check is passed, forming a closed-loop self-correction process.

[0039] The third step is closed-loop iterative optimization:

[0040] (a) Execute the test case set and generate a structural coverage report using a coverage analysis tool;

[0041] (b) Automatically evaluate whether the structure coverage report meets the coverage target defined in the domain knowledge graph and corresponding to the security level of the software under test; (c) If the coverage target is not met, automatically identify the code structure that has not been covered by the test, take the uncovered code structure information as the new test target, return to the second step, drive the large language model to generate supplementary test cases, and repeat steps (a) to (c); (d) If the coverage target is met, output the final test case set.

[0042] The aforementioned intelligent generation and closed-loop optimization system for airborne software test cases includes: a knowledge graph construction module, used to parse DO-178C standard documents and related domain documents, extract entities and relationships, and construct a domain knowledge graph that integrates DO-178C standard knowledge; a test case generation module, used to generate test cases based on the domain knowledge graph and the static analysis results of the source code of the airborne software under test, driven by a large language model (LLM), and to perform syntax verification and automatic repair on the generated test cases; and a closed-loop optimization module, connected to the test case generation module, used to execute the test cases generated by the test case generation module, generate a structure coverage report, automatically evaluate whether the structure coverage meets the preset coverage target, and when the target is not met, automatically identify the uncovered code structure and feed the uncovered information as a new test target back to the test case generation module, driving it to generate supplementary test cases until the coverage target is achieved, and output the final test case set.

[0043] Figure 2 This demonstrates the overall process of constructing a domain knowledge graph, automatically generating test cases, and verifying and supplementing them based on the DO-178C standard. The construction of the domain knowledge graph mainly involves extracting text content from PDF documents, converting it into structured data such as JSON or Markdown format, and then extracting entities and relationships to form the domain knowledge graph. At the same time, it combines other relevant specification documents to further improve the knowledge representation through entity linking and test verification processes.

[0044] For test case generation and verification, test cases are automatically generated based on LLM and checked for syntax errors. If errors are found, they are automatically fixed; if no errors are found, the process proceeds to the result verification stage. During verification, it is determined whether the test case coverage meets the DO-178C standard requirements. If not, the uncovered code is analyzed and corresponding test cases are added; if it meets the requirements, the final test case set is output, completing the automatic generation and verification process of test cases based on knowledge graph.

[0045] Figure 3It is a complete technical process for intelligent preprocessing, structured parsing and automated post-processing of documents; the process covers the whole process from raw document input to final output structured data, and has the advantages of flexibility, robustness and scalability; the system first performs the document preprocessing stage: automatically identifies the format of the input file by document type, and determines the language of the text by language recognition module, which provides the basis for subsequent processing;

[0046] The system then enters the core stage of content parsing and multimodal extraction: It comprehensively uses layout analysis technology to identify the physical and logical structure of documents, such as paragraphs, tables, and image titles, and uses the OCR enhancement module to perform high-precision character recognition on images or handwritten text. During this process, the multimodal collaboration mechanism coordinates the processing of various information formats such as text, images, and tables to achieve a unified understanding and in-depth extraction of heterogeneous document content.

[0047] After parsing, the system enters the post-processing optimization stage: sorting the extracted original content elements to reconstruct an information flow that conforms to reading logic; performing interference filtering to remove non-subject content such as headers, footers, and noisy characters; and performing semantic verification to ensure the integrity and consistency of key information and improve the quality and usability of the output results.

[0048] Ultimately, the system supports format conversion and output of the purified content as needed, generating structured data in Markdown or JSON format that is easy to integrate and process. At the same time, the system has an intermediate file retention function, which can save intermediate results during the processing according to user needs, facilitating auditing, debugging or other downstream tasks.

[0049] Figure 4 It is a complete process for knowledge graph construction based on Spacy NER model and rule matching, including three core links: entity recognition, relation extraction and entity linking. The process begins with input structured text data. First, the Spacy NER model is used to identify and extract key entities in the text. Then, the semantic relations between entities are extracted based on the predefined rule matching method, and the initial triples in the form of "entity-relation-entity" are output.

[0050] The initial triples are then processed for entity linking. First, it is determined whether the current entity is a completely new entity. If it is, a new entity node is created for it. If not, it is further determined whether there are candidate entities. If there is a uniquely matching candidate entity node, the link is made directly. If there is no candidate entity, the semantic disambiguation process is entered, and the semantic similarity between the entity and the candidate entity in the context is calculated. If the semantic similarity is higher than a set threshold, the entity is linked to the candidate entity node. If the semantic similarity is too low, a new entity node is created or a forced link is made according to the strategy. In addition, for certain references that should be attribute values ​​rather than independent entities, they will be linked to the attribute node of the corresponding entity instead of creating an independent entity.

[0051] Ultimately, all entities and relationships that have been identified, extracted, and linked are stored in the knowledge graph, completing the construction and updating of the knowledge base. The entire process effectively combines the advantages of machine learning recognition and rule matching, and ensures the uniqueness of entity nodes and the accuracy of relationships in the knowledge graph through multi-level entity disambiguation strategies.

[0052] Figure 5 It is an automated test case generation process based on the combination of static analysis and Large Language Model (LLM). The process begins with static analysis of the source code of the component to be tested, extracting requirement information from the documentation, and combining it with the constructed domain knowledge graph as input conditions for test case generation.

[0053] Then, the Large Language Model (LLM) is invoked to automatically generate test cases. After the code is generated, the test cases are checked for syntax correctness: if there are no syntax errors, they are directly output as a valid test case set; if there are syntax errors, the error information is automatically extracted, and repair prompts are constructed based on the error information and fed back to the LLM to regenerate or repair the test cases until the generated code is error-free.

[0054] Finally, the method outputs a set of valid test cases that are grammatically correct and conform to coverage criteria and test constraints, thus completing the automated test case generation process. This method effectively combines the structured information extraction capability of static analysis with the generation capability of large language models, and ensures the quality and usability of the generated test cases through syntax verification and feedback repair mechanisms.

[0055] Figure 6 The process demonstrates the verification and testing workflow for aviation software in accordance with the DO-178C standard; the workflow begins by inputting a set of generated valid test cases, dynamically executing the software under test, and calculating test coverage in real time.

[0056] The test results are then automatically reviewed and bidirectional traceability verification is performed. The system automatically analyzes and generates trace data to ensure that each software requirement has at least one test case coverage, tracing back from requirement to test; each test case can be traced back to one or more software requirements, tracing back from test to requirement; if any missing trace links are found during the verification process, the system will automatically generate a non-conformity report and guide the correction to ensure that the final delivered trace data is complete and compliant; at the same time, it determines whether the current coverage meets the requirements of the DO-178C standard: if it meets the requirements, the verification passes and a final coverage report is generated; if it does not meet the requirements, the uncovered code is analyzed and the specific uncovered targets are located, and corresponding test cases are added accordingly.

[0057] Ultimately, through iterative supplementation of test cases and coverage assessment, the software testing is ensured to fully meet the high security requirements of DO-178C, outputting a complete coverage report required for certification and completing the entire verification process. This method combines dynamic testing, coverage statistics, requirement traceability, and automatic review, effectively ensuring the reliability and compliance of aerospace-grade software.

Claims

1. A method for intelligent generation and closed-loop optimization of airborne software test cases, characterized in that, Includes the following steps: (1) Knowledge graph construction: Analyze the DO-178C standard document and related domain documents, extract the entities and relationships defined therein, and construct a domain knowledge graph that integrates the knowledge of the DO-178C standard; (2) Initial test case generation: Based on the domain knowledge graph and combined with the static analysis results of the source code of the airborne software under test, an initial test case set is generated using a large language model. The generation process includes performing syntax verification on the generated test cases, and if the verification fails, the error information is fed back to the large language model to trigger automatic repair until the verification is passed. (3) Closed-loop iterative optimization: (a) Execute the test case set and generate a structural coverage report using the coverage analysis tool; (b) Automatically evaluate whether the structural coverage report meets the coverage target defined in the domain knowledge graph and corresponding to the security level of the software under test; (c) If the coverage target is not met, automatically identify the code structure that has not been covered by the test, take the uncovered code structure information as the new test target, return to step (2), drive the large language model to generate supplementary test cases, and repeat steps (a) to (c); (d) If the coverage target is met, output the final test case set.

2. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 1, characterized in that, Step (1) includes: using a document parsing tool to extract and structure the text of the DO-178C standard document; Natural language processing technology is used to perform entity recognition and relation extraction on structured text, and triples containing concepts such as requirements, test cases, coverage criteria, traceability data and their relationships are constructed. By using entity linking and disambiguation techniques, triples are stored in a graph database to form the domain knowledge graph.

3. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 1, characterized in that, Step (2) of generating test cases using a large language model includes: Based on the domain knowledge graph and static analysis results, normal range test case prompts for verifying the expected behavior of the software and robustness test case prompts for verifying the behavior of the software under abnormal input and boundary conditions are generated. The normal range test case prompts and robustness test case prompts are input into the large language model to generate initial test cases covering normal functions and exception handling.

4. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 3, characterized in that, The design of the normal range test cases and the robustness test cases conform to the DO-178C standard.

5. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 1, characterized in that, In the closed-loop iterative optimization of step (3), after identifying the code structure that has not been covered by the test, the uncovered code structure is automatically and backward traced back to the low-level requirements it implements; the low-level requirements are combined with the uncovered code structure information to form a test target with requirement traceability for generating supplementary test cases.

6. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 1, characterized in that, The structure coverage target is statement coverage, branch coverage, or modified condition / decision coverage.

7. The intelligent generation and closed-loop optimization method for airborne software test cases according to claim 6, characterized in that, The software under test has the highest security level, and the target structure coverage is the corrected condition / decision coverage.

8. An intelligent generation and closed-loop optimization system for airborne software test cases, characterized in that, include: Knowledge graph construction module; Test case generation module; Closed-loop optimization module.

9. The intelligent generation and closed-loop optimization system for airborne software test cases according to claim 8, characterized in that, The test case generation module uses the domain knowledge graph and static analysis results to generate normal range test case prompts for verifying the expected behavior of the software and robustness test case prompts for verifying the behavior of the software under abnormal input and boundary conditions, respectively, and inputs the prompts into a large language model.

10. The intelligent generation and closed-loop optimization system for airborne software test cases according to claim 8, characterized in that, The closed-loop optimization module will identify uncovered code structures and automatically trace back to their corresponding lower-level requirements, combining the requirements and code structure information into new test targets, and feeding them back to the test case generation module.

Images