SCADE test case automatic generation method based on large language model

Abstract

The invention discloses an automatic SCADE test case generation method based on a large language model, which comprises the following steps of: analyzing various SCADE related documents to obtain an analysis result; the analysis result is converted into JSON data; performing key information extraction on the JSON data, and performing block processing on the extracted key information to obtain text blocks; vectorizing the text blocks, storing the vectorized text blocks into a vector database, converting a natural language query requirement of a user into a query vector, and performing approximate nearest neighbor search with the query vector in the vector database by utilizing RAG retrieval to obtain a preliminary retrieval result; based on metadata in the preliminary retrieval result, filtering, timeliness sorting and deduplication fusion processing are carried out, and a retrieved context information set is obtained; adopting a dynamic cue word template to construct an enhanced cue word based on the retrieved context information set and the natural language query of the user; and calling a large language model API, inputting an enhanced cue word, and generating a structured test case.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology in safety-critical areas such as avionics and rail transportation, and specifically relates to a method for automatically generating SCADE test cases based on a large language model. Background Technology

[0002] A safety-critical system is a system that, upon failure or malfunction, could cause serious personal injury, widespread environmental degradation, substantial property damage, and delays or failures of critical tasks. These systems are widely used in critical fields such as aerospace, rail transportation, nuclear power, and chemical engineering. Safety-critical software, as a component of a safety-critical system, refers to software used within that system that may lead to unacceptable risks. To develop safety-critical software, relevant industries have proposed specific software safety engineering standards, including the airworthiness standard DO-178C, the EN 50128 standard for the railway industry, the IEC 61508 standard for the electronics and electrical industries, and the ISO 26262 standard for the automotive industry. Safety, reliability, and real-time performance are fundamental requirements for these systems and software.

[0003] As the scale and complexity of safety-critical software continue to increase, Model-Based Development (MBD) methods are widely used in safety-critical fields. SCADE is a safety-critical software design tool based on MBD, widely used in fields with high reliability requirements such as avionics and rail transportation. Its core is to describe system behavior through formal models (such as data flow diagrams and state machines) and automatically convert them into executable code through code generation technology.

[0004] Software testing plays a crucial role in ensuring software quality and improving software reliability, and is also a key technology widely used in industry. Testing can uncover many hidden defects in software. However, the generation of test cases for SCADE models still relies on manual design or automated tools based on simple rules, which has the following limitations: manual design is prone to overlooking boundary conditions or abnormal scenarios, resulting in insufficient coverage; complex models require a significant amount of time to write test specifications, leading to low design efficiency; testers need dual experience in SCADE modeling and safety standards (such as DO-178C, EN 50128), heavily relying on domain knowledge; and during the testing process, it is difficult to clearly, accurately, and completely establish the correspondence between each set of test cases and specific clauses in the requirements specification, resulting in poor traceability between test cases and requirements specifications.

[0005] In existing technologies, rule-based test case generation methods struggle to handle the complex logic of SCADE models, while pure large-model generation methods are prone to generating invalid or dangerous test cases due to a lack of domain constraints. Therefore, there is an urgent need for an automated test case generation solution that combines domain knowledge enhancement with large-model reasoning capabilities. Summary of the Invention

[0006] Purpose of the invention: This invention proposes an automatic SCADE test case generation method based on a large language model. It aims to explore the use of large language models to assist in the automatic generation of test cases for safety-critical software SCADEs, and solve the problem that manual or rule-based test generation methods are difficult to handle the complex logic of SCADE models, thereby improving the efficiency, quality and coverage of model testing.

[0007] Technical solution: A method for automatically generating SCADE test cases based on a large language model, comprising the following steps: Step 1: Parse various SCADE-related documents to obtain the parsing results; convert the parsing results into JSON data; Step 2: Extract key information from the JSON data, divide the extracted key information into blocks to obtain text blocks; vectorize the text blocks and store them in a vector database, where each vector data record in the vector database includes a vector, the original text, and metadata; Step 3: Convert the user's natural language query into a query vector, use RAG retrieval to perform a near nearest neighbor search on the vector database to obtain preliminary search results; based on the metadata in the preliminary search results, perform filtering, timeliness sorting and deduplication fusion processing to obtain the retrieved context information set. Step 4: Using dynamic suggestion word templates, construct enhanced suggestion words based on the retrieved context information set and the user's natural language query; Step 5: Call the Large Language Model API, input the enhanced prompt words, and generate structured test cases.

[0008] Furthermore, it also includes the following steps: Step 6: Convert the test cases into SCADE test sequences, drive the simulator to execute them, evaluate the execution results, and trigger confirmation output based on the evaluation results or perform feedback optimization in Step 7. Step 7: Take the unsatisfactory evaluation results and their root causes as input, pass them to the prompt word optimizer, drive it to adjust the RAG search strategy or prompt word template parameters, and restart the generation process that started from Step 5 until the output results pass the evaluation.

[0009] Furthermore, in step 1, the parsing of various SCADE-related documents to obtain parsing results specifically includes: The SCADE-related documents include SCADE project documents, requirements design documents, and historical test case documents. Parse the .etp, .scade, and .xscade files in the SCADE project documentation to extract the project structure, input / output, type settings, value range information, operator interfaces, state machine logic, and module call relationships. Natural language processing technology is used to parse the requirements design document and extract test-related entities, condition-action pairs, and interface definitions. The historical test case document includes historical test cases and MC / DC coverage reports.

[0010] Furthermore, in step 2, the extraction of key information from the JSON data, followed by segmentation of the extracted key information into text blocks, specifically includes: Based on a predefined rule template, key information is extracted from the JSON data. The key information includes: requirement body, function description, interface definition, state machine state set, state machine transition conditions, and security integrity level. The interface definition includes: input and output variables, data types, and ranges. The extracted key information is divided into logical units to ensure that a single state machine, data flow node, or complete requirement clause is kept in the same text block. A recursive character segmentation method is used to perform secondary segmentation on text blocks whose size is larger than a set value.

[0011] Furthermore, in step 2, the metadata includes at least the source file name, chapter number, and content type.

[0012] Furthermore, in step 3, the process of converting the user's natural language query request into a query vector, and then using RAG retrieval to perform a near-nearest neighbor search in the vector database to obtain preliminary search results, specifically includes: Based on the key elements of the SCADE domain dictionary and test case generation task, the user's natural language query requirements are extended to enhanced query requirements rich in keywords; the key elements of the test case generation task include: function name, input and output variables, state machine, transition conditions, and requirement identifier. Transform enhanced query requirements into query vectors; An approximate nearest neighbor search request is initiated to the vector database. The vector database calculates the cosine similarity between the query vector and all vectors in the database, and returns the original text blocks corresponding to the Top-K vectors with the highest similarity, forming preliminary search results.

[0013] Furthermore, in step 3, the metadata in the preliminary search results is filtered, sorted by timeliness, and deduplicated and merged to obtain the retrieved context information set, specifically including: Apply content type-based filters for filtering; If the metadata in the initial search results contains version information, the document with the latest version number will be returned first. Redundant results from the same original section (section_id) or describing the same functional module are deduplicated, and semantic fusion is performed on text blocks with similar content; Finally, the retrieved context information set is obtained, which includes SCADE model knowledge, requirement terms, and interface definitions most relevant to the current test case generation task.

[0014] Furthermore, in step 4, the use of dynamic prompt word templates to construct enhanced prompt words based on the retrieved context information set and the user's natural language query specifically includes: Clearly define the roles and tasks of the large model in the dynamic prompt word template; The context information set is categorized according to the content_type field in its metadata, and injected in a categorized manner using explicit tags to form the basis for task generation; The dynamic prompt word template incorporates a chain-thinking instruction, which requires the model to execute the following steps: Please follow these steps to think about and generate test cases: Analysis requirements: The analysis requirements include: understanding the functional, performance, and security requirements specified in the user's natural language query; Identification Interface: The identification interface includes: clearly defining all input and output variables of the system under test, their data types, and value ranges; Define test scenarios: The definition of test scenarios includes designing various test scenarios, including normal, boundary, and abnormal scenarios, based on requirements and interfaces; Write specific test cases for each test scenario; The dynamic prompt word template includes output format instructions; these instructions require the model to generate test case files with the .sss extension, indicating that: SSM::set <var> <val> SSM::cycle[ <integer>] Here, var refers to the name of the variable to be assigned, val refers to the specific value assigned to the var variable, and cycle refers to the number of cycles in which this abstract test case is run; Negative constraints and quality requirements are imposed on the generation of test cases; these include: it is strictly forbidden to fabricate or invent any requirements, interfaces or functions not mentioned in the provision context; the generated test cases must be repeatable; all declared exceptions and boundary conditions must be covered; and each test case must be able to be explicitly traced back to at least one requirement ID.

[0015] Furthermore, in step 6, the process of converting test cases into SCADE test sequences, driving the simulator to execute them, evaluating the execution results, and triggering confirmation output or feedback optimization in step 7 based on the evaluation results specifically includes: Convert test cases into SCADE test sequences to drive the simulator to execute them; The pass rate is calculated based on the execution results; the pass rate refers to the percentage of test cases that are successfully loaded and fully executed by the SCADE environment out of the total number of test cases. Test cases that cannot be loaded or executed will be immediately marked and categorized by the system. The execution results are used to statistically analyze whether simulation anomalies were triggered during execution, in order to verify whether the expected dynamic behavior of the model in the test cases matches the actual behavior. Calculate the requirement coverage rate based on the execution results; the requirement coverage rate refers to the percentage of requirement items verified by the test cases that have passed the tests out of the total number of all relevant requirement items; Calculate branch coverage and MC / DC coverage based on the execution results; branch coverage refers to the percentage of covered model logic branches out of all branches; The calculated indicators are compared with the preset quality thresholds. If all indicators meet the standards, the generated test cases are confirmed as valid outputs after the evaluation process is completed. If any indicator fails to meet the standards, step 7 is executed for feedback optimization.

[0016] Furthermore, the process of taking the substandard evaluation results and their root causes as input to the prompt word optimizer drives it to adjust the RAG retrieval strategy or prompt word template parameters, and restarts the generation process starting from step 5 until the output results pass the evaluation. Specifically, this includes: By analyzing the prompts and RAG search results used when generating this use case, we traced back to the text blocks and metadata in the vector database that provided the source context to locate the source of the erroneous knowledge. If the problem is determined to be due to a lack of knowledge, the missing correct knowledge, including the latest design documents and revised requirements specifications, will be regenerated into vector data records and added to the vector database. If the problem is determined to be due to knowledge errors or obsolescence, the corresponding text block records in the vector database are marked, including adding the deprecated tag to their metadata, reducing the weight of the record during the retrieval process, or completely removing it from the database. Continuously monitor and record the evaluation metrics of the test cases generated under each prompt word template version, and establish a quantitative mapping relationship between prompt word design and generation effect; To address specific performance shortcomings, different prompt word optimization strategies are deployed in parallel for A / B testing, and a new template version with better performance is selected based on the test data. Version control will be implemented for the optimized prompt word templates.

[0017] Beneficial Effects: This invention discloses an automatic test case generation method for SCADE based on a large language model, belonging to the field of safety-critical system model verification technology. The method first performs multi-source parsing and structuring processing on SCADE project documents, requirement specifications, and historical test cases, and constructs a vector knowledge base rich in metadata. By receiving user natural language queries, it accurately retrieves relevant context using a semantic retrieval and metadata filtering fusion strategy. Combined with dynamically constructed multi-level prompt word templates, it integrates system role definitions, retrieval context, chain-like reasoning guidance, structured output format, and negative constraints to accurately guide the large language model to generate test cases that meet engineering requirements. Subsequently, it drives the SCADE simulation environment to automatically execute and quantitatively evaluate the generated test cases, calculating key indicators such as pass rate, requirement coverage, and MC / DC coverage. Finally, a feedback loop is established based on the evaluation results to drive iterative optimization of the knowledge base and prompt word templates. This invention, through intelligent test case generation and verification, effectively solves the technical problems of low efficiency and difficulty in covering complex logical conditions in traditional methods while ensuring high coverage and safety standard compliance. Compared with existing technologies, this invention has the following advantages: (1) This invention is the first to apply large models to the test generation of SCADE safety-critical modules, filling the gap in the traditional method in complex logic coverage, solving the strict requirements of high-difficulty coverage such as MC / DC in the field of safety-critical systems, and providing strong technical support for the development of safety-critical systems such as avionics and automotive electronics.

[0018] (2) High coverage and accuracy: RAG retrieval ensures domain knowledge constraints, prompt word engineering guides the generation of standard test cases from large models, and proposes a closed-loop optimization framework of "simulation feedback + feedback iteration" to improve the quality of test case generation.

[0019] (3) Efficiency improvement: The automated generation time is significantly shorter than that of manual generation. It is suitable for large-scale SCADE model libraries and has the ability to continuously learn and self-optimize. It can continuously improve the generation effect as the usage process deepens, and has important engineering application value and promotion prospects.

[0020] (4) Explainability and controllability: The layered prompt word template makes the generation process transparent, which is convenient for security certification agencies to review, and has a complete automated evaluation and feedback optimization system. Attached Figure Description

[0021] Figure 1 This is an architecture diagram of an automatic SCADE test case generation method based on a large language model proposed in an embodiment of the present invention, showing the interaction logic between the seven steps; Figure 2 The flowchart below shows a method for automatically generating SCADE test cases based on a large language model, as proposed in an embodiment of the present invention, illustrating SCADE document processing. Figure 3 This is a word segmentation structure diagram of the prompt word template proposed in an embodiment of the present invention. Detailed Implementation

[0022] The objects and functions of the present invention, as well as the methods for achieving these objects and functions, will be clarified by referring to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below; it can be implemented in various forms. The purpose of this specification is merely to help those skilled in the art to comprehensively understand the specific details of the invention.

[0023] like Figure 1 As shown in the figure, this embodiment of the invention proposes an automatic SCADE test case generation method based on a large language model, including the following steps: Step 1: Process the input SCADE-related documents, preprocessing and denoising them, parse the SCADE model, and convert the parsing results into JSON format. See also... Figure 2 .

[0024] Sub-step 1: Parse the .etp, .scade, and .xscade files using the SCADE Suite SDK API to obtain the project structure and extract model file paths, dependencies, and constant type definitions. Identify... <project> 、 <model>Tags are used to extract key information, parse the input / output ports, value range information, data types and expressions of operators, and record the call relationships between modules, such as the main module calling a submodule. Graphic style information in .scade files is filtered, retaining only the logic part; for .xscade files, data flow, state machine transition conditions, and action logic are extracted, ignoring layout coordinates.

[0025] Sub-step 2: Use Natural Language Processing (NLP) to parse the model's requirements and design documents, extracting plain text and removing irrelevant information such as headers, footers, and comments. Parse key requirement descriptions in the SCADE model design document, such as extracting test-related entities like input ranges and expected outputs, identifying condition-action pairs, state machine descriptions and transition conditions, and interface definitions.

[0026] Sub-step 3: Load historical test cases (.sss files) and the MC / DC coverage report to build the initial test case set for this model. The .sss file is the test case format used by SCADE Test Integrator. It contains specific input stimulus settings, output expectation checks, and execution commands (such as SSM::set, SSM::check, SSM::cycle). The large model can learn from these historical test cases to accurately master the format, syntax, and writing style of test cases. This makes the newly generated test cases recognizable and executable by the SCADE testing environment, and engineering-ready, greatly reducing subsequent format conversion and debugging work.

[0027] Sub-step 4: After parsing the various documents in SCADE, the parsing results are obtained, generating a structured representation. This representation is then converted into a JSON file for storage and display, for example: { "operator_name": "AddOperator", "inputs": ["a: int8", "b: int8"], "outputs": ["result: int8"], "logic": "result = a + b", "constraints": ["a>= 0", "b<= 100"] } Step 2: Extract key information and remove noise from the JSON data. Use a strategy that combines primary and secondary segmentation to segment the content of the SCADE model. Use a dedicated embedding model to vectorize the text blocks and store them in a vector database. Each record contains vectors, original text and metadata.

[0028] Sub-step 1: Preprocess the JSON data passed in after step 1, including: Key information extraction: Based on predefined rule templates, focus on extracting fields that are strongly related to test case generation, including but not limited to: requirement_text (requirement text), function_description (functional description), interface_definition (interface definition, including input and output variables, data types, and ranges), state_machine_states (state machine state set), state_machine_transitions (state machine transition conditions), and safety_integrity_level (safety integrity level).

[0029] Noise removal: Remove redundant information that is irrelevant to the functional logic, such as version history, author information, formatting tags, etc.

[0030] Sub-step 2: Implement a block-based strategy for the SCADE model content, dividing it into blocks according to operators and state machines. Each block contains logical descriptions and constraints. Considering the highly structured and modular nature of the SCADE model, two block-based strategies are implemented: Primary segmentation (logical unit division): The algorithm prioritizes the use of the natural boundaries inherent in the JSON structure (such as section_id, node_name, state_machine_id) for the first level of segmentation, ensuring that a single state machine, data flow node, or complete requirement clause is kept in the same text block as much as possible.

[0031] Secondary segmentation (recursive segmentation): For logical units that are too large, a recursive character segmentation method is used. The key parameters of this invention are set as follows: chunk_size: 512 tokens. This length balances the performance of the embedding model with the amount of contextual information, and is sufficient to accommodate multiple interface variable definitions or a complete state transition rule. chunk_overlap: 80 tokens. Setting an overlap area aims to prevent key semantic information, such as the latter part of a transition condition, from being fragmented into different blocks, ensuring the contextual coherence of the retrieval results and improving the accuracy of subsequent test case generation.

[0032] Sub-step 3: Vectorization and storage, transforming textual knowledge into numerical vectors and constructing a vector database that supports efficient similarity retrieval.

[0033] Embedding Model Selection: To accurately capture the formal semantic and syntactic features of the SCADE model, this invention selects the professional embedding model BAAI / bge-base-en-v1.5, pre-trained on code, mathematical, and technical document corpora. This model excels at understanding logical structures, operators, variable types, and state transitions in technical languages, and its generated vector representations significantly outperform general-domain models in semantic fidelity.

[0034] Vectorization process: For each text block Ci after block processing, it is input into the selected embedding model E for encoding, generating a fixed high-dimensional numerical representation: Vi=E(Ci), where Vi is a 768-dimensional floating-point vector, which is the semantic embedding of the text block Ci in the vector space.

[0035] Vector Database Construction: The generated vectors and source data are stored in the high-performance vector database ChromaDB (SCADE model specification documents, historical test cases, and security standard clauses such as DO-178C Objective 6) to support millisecond-level approximate nearest neighbor (ANN) searches. Each record in the database contains the following fields, forming a complete searchable unit:

[0036] Step 3: The RAG retrieval module receives user queries and accurately retrieves the contextual information most relevant to the test case generation task from the vector database.

[0037] Sub-step 1: Query Reception and Enhancement: This module receives query requests Q input by the user in natural language (e.g., "Generate test cases for the 'emergency braking' function in the 'brake control system'"). To improve retrieval recall, the system integrates a lightweight query understanding component. This component automatically expands the original query Q into a keyword-rich enhanced query Q' based on the SCADE domain dictionary and key elements of the test case generation task. These key elements include, but are not limited to: function name, input / output variables, state machine, transition conditions, and requirement identifiers.

[0038] Sub-step 2: Semantic Retrieval Execution: The system uses the same embedding model E as in the vectorization process to convert the enhanced query Q' into a query vector V_q, which is calculated as: V_q = E(Q'). Subsequently, the retrieval module initiates an Approximate Nearest Neighbor (ANN) search request to the vector database. The vector database utilizes its pre-built efficient index structure to calculate the cosine similarity between the query vector V_q and all vectors in the database, and returns the original text blocks corresponding to the Top-K vectors with the highest similarity, forming a preliminary retrieval result list.

[0039] Sub-step 3: Metadata-based post-processing and reordering aims to address the issue of irrelevant information potentially introduced by pure vector search. After receiving the preliminary results, the retrieval module does not return them directly but performs the following intelligent post-processing: Metadata Filtering: The system can automatically apply metadata filters based on predefined strategies. For example: `filter ={"content_type": {"$in": ["requirement", "interface"]}}` This command instructs the database to "prioritize returning blocks with content types 'requirement' or 'interface' from the results with the highest similarity." This ensures that the search results closely match the inputs, outputs, and requirement specifications needed for test case generation.

[0040] Timeliness ranking: If the metadata contains version information, the system will prioritize returning the document content with the latest version number to ensure that the knowledge on which the test cases are based is up-to-date.

[0041] Deduplication and semantic fusion: Redundant results from the same original chapter (section_id) or describing the same functional module are deduplicated, and text blocks with similar content are semantically fused to generate coherent and concise contextual information, avoiding information overload.

[0042] After the above processing, the RAG retrieval module finally outputs a filtered, sorted, and refined set of context information, denoted as Context = [doc1, doc2, ..., docK]. This set contains the SCADE model knowledge, requirement clauses, and interface definitions most relevant to the current test case generation task, providing a precise domain knowledge foundation for the subsequent prompt word engineering module to build high-quality generation instructions.

[0043] Step 4: Use the prompting engineering module to receive refined context from the RAG retrieval module and build structured, high-quality prompt words to accurately guide the large language model in generating compliant test cases.

[0044] Sub-step 1: The module consists of two parts. The first is the output of step 3, which is an ordered list of sorted and filtered SCADE knowledge fragments most relevant to the user's query [doc1, doc2, ..., docK]. The second is the natural language test generation request submitted by the user, such as: "Generate test cases for the 'normal operation' status of the 'main control module'".

[0045] Sub-step 2: Cue word construction: A predefined, dynamically populated cue word template is used. This template is a multi-part structured text used to organically integrate system instructions, domain knowledge, and task constraints. For example... Figure 3 As shown, its construction method is as follows: System Role Definition: First, clearly define the roles and tasks of the large model in the template. For example: "You are a senior safety engineer specializing in avionics and automotive electronics, proficient in SCADE development processes and safety standards such as DO-178C / ISO26262. Your task is to strictly follow the provided design documents and requirements to generate high-quality, traceable SCADE test cases that cover all model paths, have high MC / DC coverage, and are reliable." Task Context Injection: Subsequently, the text blocks in the input Context list are categorized according to the `content_type` field in their metadata, and injected using explicit tags (such as `## Related Requirements##`, `## Interface Definition##`, `## State Machine Model##`) to form the main basis for task generation, thereby avoiding information chaos. Example: Please generate test cases strictly according to the following design information. ## Related Requirements ##: {requirement_chunk_1} {requirement_chunk_2}” ## System Interface Definition ##: {interface_chunk_1}” State Machine Model {state_machine_chunk_1}” Sub-step 3: To logically decompose the complex test case generation task, a chain-of-thought instruction is embedded in the prompts, forcing the model to perform step-by-step reasoning. This instruction requires the model to execute the following steps: Please follow these steps to think about and generate test cases: Analyze the requirements: First, understand the functional, performance, and security requirements specified in the above requirements.

[0046] Identify the interface: Secondly, clearly define all input and output variables of the system under test, their data types, and value ranges.

[0047] Define test scenarios: Next, based on the requirements and the analyzed interfaces, design various test scenarios, including normal, boundary, and abnormal ones.

[0048] Generate specific test cases: Finally, write specific test cases for each test scenario. Sub-step 4: To ensure the generated results can be directly integrated into downstream testing toolchains, the prompt includes strict output format instructions. These instructions require the large model to output content in a specified structured format: the .sss file format recognized by the SCADE test suite. The instructions include specific examples and explicitly stipulate that the output must include key fields and syntax such as input stimulus settings (SSM::set), output expectation checks (SSM::check), and simulation cycle control (SSM::cycle).

[0049] Example: ## Output Format Requirements ## You must output in a strict format, including all input and output fields: ## inputs # int8 SSM::set Add / Input1 4 # int8 SSM::set Add / Input2 5 ## outputs # int8 SSM::check Add / Output1 9 SSM::cycle Sub-step 5: Impose negative constraints and quality requirements on the generation of test cases, explicitly prohibit certain behaviors in the model, and propose high-quality requirements to avoid common defects in large models.

[0050] ## Important Notes ## It is strictly forbidden to fabricate or invent any requirements, interfaces, or functions that are not mentioned in the context of the provision.

[0051] The generated test cases must be repeatable.

[0052] All declared exceptions and boundary cases must be covered.

[0053] Each test case must be able to be explicitly traced back to at least one requirement ID. Through the above sub-steps, the prompting engineering module outputs an enhanced prompt word that is structurally rigorous, information-rich, and clearly constrained, providing a reliable guarantee for the subsequent generation of high-precision, high-quality, and directly usable test cases for large models.

[0054] Step 5: By calling the APIs of large language models such as OpenAI GPT-4, Meta LLaMA, and Deepseek, the enhanced prompt words generated by the prompt engineering module described in Step 4 are sent as input to the API. The large language model performs inference calculations based on the received instructions, context, and constraints, and outputs the generated text content.

[0055] Successful outcome: The model outputs a structured data object that conforms to the specified format. This object fully contains test input stimuli, expected outputs, simulation commands, and requirement traceability information. The system serializes this data object into a .sss file and outputs it to the specified project file directory (e.g., [Project_Root] / TestCases / ). This file can be directly displayed to the user for review or automatically imported into the SCADE Test testing environment and test management tools for execution in subsequent processes.

[0056] Abnormal situations: The system captures and outputs clear error logs and status codes. Typical anomalies include output format parsing failure, inclusion of variables or logic not defined in the provided context, failure to meet chained thinking steps or negative constraints, etc. The system automatically sends the failed generated results, corresponding enhanced prompts, and error messages to the "manual review queue" for final judgment, correction, and approval by humans. All cases entering the manual review queue are recorded to form historical data, which is used for subsequent iterative optimization of prompt templates and search strategies.

[0057] Step 6: Automated execution and quantitative evaluation of the generated test cases, calculate pass rate, runtime errors, requirement coverage, and branch MC / DC coverage metrics to objectively measure their quality and ensure they meet the requirements of safety-critical standards.

[0058] Sub-step 1: The system automatically converts the .sss test case file generated in step 5 into an input format recognized by the SCADE test suite by calling the application programming interface (API) provided by Ansys SCADE. Subsequently, the system drives the SCADE Simulator or formal verification tool in the background to execute the test sequence and automatically collects the execution logs, error reports, and coverage data generated during the simulation process.

[0059] Sub-step 2: After execution, the system analyzes the collected data and calculates the following key quality indicators: Pass rate: The percentage of test cases that are successfully loaded and fully executed by the SCADE environment out of the total number of test cases. Test cases that fail to load or execute, such as those with syntax errors or misspelled variable names, will be immediately marked and categorized by the system.

[0060] Runtime errors: This metric tracks whether simulation exceptions such as division by zero, array out-of-bounds errors, and numerical overflows occur during execution. It is used to verify whether the expected dynamic behavior of the model matches the actual behavior as predicted by the test cases.

[0061] Requirements coverage: This is the percentage of requirements verified by test cases that have passed testing, out of all relevant requirements. This metric is a mandatory requirement for meeting functional safety standard certification.

[0062] Branch coverage and MC / DC coverage: By analyzing the detailed coverage report generated by the SCADE toolchain, the percentage of covered model logic branches out of all branches is calculated. For the highest security integrity level (such as DO-178C Level A), the system focuses on calculating modified MC / DC coverage, which is the core basis for objectively evaluating the depth of test cases for testing complex logic conditions.

[0063] Sub-step 3: The system compares the calculated indicators with the preset quality thresholds.

[0064] If all indicators meet the standards, the generated test cases are confirmed as valid outputs after the evaluation process is completed.

[0065] If any indicator fails to meet the standard, the system automatically triggers a feedback optimization mechanism. This mechanism takes the evaluation result of the failure and its root cause as input, passes it to the prompt word optimizer, drives it to adjust the RAG retrieval strategy or prompt word template parameters, and restarts the generation process starting from step 5 until the output result passes the evaluation.

[0066] Step 7: Feedback iteration, a feedback loop from verification results to generation basis, drives the self-evolution of knowledge base and generation strategy through analysis and evaluation results, so as to continuously improve the generation quality of test cases.

[0067] Sub-step 1: Perform root cause analysis on the use cases that expose problems in the simulation and optimize the knowledge base. If the problem stems from a lack of knowledge or the retrieval of incorrect information, the knowledge base update process is triggered.

[0068] Defect Root Cause Analysis and Tracing: When the evaluation in step 6 indicates that the test case has a defect, the system analyzes the prompt words used when generating the test case and the RAG search results, and traces back to the text block and its metadata that provide the source context in the vector database to locate the knowledge source that may have caused the error.

[0069] Knowledge Enhancement: If the problem is determined to be due to a lack of knowledge, such as a lack of definition for a certain abnormal condition, the missing correct knowledge, including the latest design documents and the revised requirements specifications, is re-entered into the document processing module described in step 1 for parsing, segmentation, and vectorization, and the generated new vector data records are added to the vector database.

[0070] Knowledge De-weighting and Removal: If the problem is determined to be due to knowledge errors or outdated knowledge, such as changes in the interface definition, the system will mark the corresponding text block records in the vector database. This includes adding the deprecated tag to its metadata, de-weighting the record during the retrieval process to reduce its retrieval priority, or completely removing it from the database.

[0071] Sub-step 2: Establish a mapping relationship between evaluation results and prompt word effectiveness, iterate prompt words based on data-driven approaches, and continuously optimize prompt word templates.

[0072] Performance monitoring and mapping establishment: The system continuously monitors and records the evaluation metrics of the test cases generated under each prompt word template version, and establishes a quantitative mapping relationship between prompt word design and generation effect.

[0073] A / B Testing and Strategy Iteration: For specific performance bottlenecks (such as "low MC / DC coverage"), the system deploys different prompt word optimization strategies in parallel (such as strengthening the instruction "cover all logical branches" in the template) for A / B testing, and selects a new template version with better performance based on the test data.

[0074] Version Management and Deployment: Strict version control is implemented for the optimized prompt word templates. Once a new template is verified as valid in step 6, the system automatically upgrades it to the current default template, completing the iterative update of the prompt word generation strategy.

[0075] Sub-step 3: Generate test report Comprehensive Assessment Report: Generate a structured report that details all quantitative indicators of this assessment (feasibility, requirement coverage, model logic coverage), the problems found, and their root causes.

[0076] Targeted optimization suggestions: Output specific and actionable actions, such as: "It is recommended to add boundary value test cases for requirement REQ_ID_xxx" or "Ambiguity was detected in the definition of the value range of variable SensorValue. It is recommended to check and update the interface documentation."< / model> < / project> < / integer> < / val> < / var>

Claims

1. A method for automatically generating SCADE test cases based on a large language model, characterized in that, Includes the following steps: Step 1: Parse various SCADE-related documents to obtain the parsing results; convert the parsing results into JSON data; Step 2: Extract key information from the JSON data, divide the extracted key information into blocks to obtain text blocks; vectorize the text blocks and store them in a vector database, where each vector data record in the vector database includes a vector, the original text, and metadata; Step 3: Convert the user's natural language query into a query vector, use RAG retrieval to perform a near nearest neighbor search on the vector database to obtain preliminary search results; based on the metadata in the preliminary search results, perform filtering, timeliness sorting and deduplication fusion processing to obtain the retrieved context information set. Step 4: Using dynamic suggestion word templates, construct enhanced suggestion words based on the retrieved context information set and the user's natural language query; Step 5: Call the Large Language Model API, input the enhanced prompt words, and generate structured test cases.

2. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, It also includes the following steps: Step 6: Convert the test cases into SCADE test sequences, drive the simulator to execute them, evaluate the execution results, and trigger confirmation output based on the evaluation results or perform feedback optimization in Step 7. Step 7: Take the unsatisfactory evaluation results and their root causes as input, pass them to the prompt word optimizer, drive it to adjust the RAG search strategy or prompt word template parameters, and restart the generation process that started from Step 5 until the output results pass the evaluation.

3. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, Step 1 involves parsing various SCADE-related documents to obtain the parsing results; specifically, this includes: The SCADE-related documents include SCADE project documents, requirements design documents, and historical test case documents. Parse the .etp, .scade, and .xscade files in the SCADE project documentation to extract the project structure, input / output, type settings, value range information, operator interfaces, state machine logic, and module call relationships; Natural language processing technology is used to parse the requirements design document and extract test-related entities, condition-action pairs, and interface definitions. The historical test case document includes historical test cases and MC / DC coverage reports.

4. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, In step 2, key information is extracted from the JSON data, and the extracted key information is divided into blocks to obtain text blocks. Specifically, it includes: Based on a predefined rule template, key information is extracted from the JSON data. The key information includes: requirement body, function description, interface definition, state machine state set, state machine transition conditions, and security integrity level. The interface definition includes: input and output variables, data types, and ranges. The extracted key information is divided into logical units to ensure that a single state machine, data flow node, or complete requirement clause is kept in the same text block. A recursive character segmentation method is used to perform secondary segmentation on text blocks whose size is larger than a set value.

5. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, In step 2, the metadata includes at least the source file name, chapter number, and content type.

6. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, Step 3, which involves converting the user's natural language query into a query vector and using RAG retrieval to perform a near-nearest neighbor search in the vector database to obtain preliminary search results, specifically includes: Based on the key elements of the SCADE domain dictionary and test case generation task, the user's natural language query requirements are extended to enhanced query requirements rich in keywords; the key elements of the test case generation task include: function name, input and output variables, state machine, transition conditions, and requirement identifier. Transform enhanced query requirements into query vectors; An approximate nearest neighbor search request is initiated to the vector database. The vector database calculates the cosine similarity between the query vector and all vectors in the database, and returns the original text blocks corresponding to the Top-K vectors with the highest similarity, forming preliminary search results.

7. The method for automatically generating SCADE test cases based on a large language model according to claim 6, characterized in that, In step 3, the metadata in the preliminary search results is filtered, sorted by timeliness, and deduplicated and merged to obtain the retrieved context information set, specifically including: Apply content type-based filters for filtering; If the metadata in the initial search results contains version information, the document with the latest version number will be returned first. Redundant results from the same original chapter or describing the same functional module are deduplicated, and text blocks with similar content are semantically fused. Finally, the retrieved context information set is obtained, which includes SCADE model knowledge, requirement terms, and interface definitions most relevant to the current test case generation task.

8. The method for automatically generating SCADE test cases based on a large language model according to claim 1, characterized in that, In step 4, the use of dynamic prompt word templates, based on the retrieved context information set and the user's natural language query, constructs enhanced prompt words, specifically including: Clearly define the roles and tasks of the large model in the dynamic prompt word template; The context information set is categorized according to the content_type field in its metadata, and injected in a categorized manner using explicit tags to form the basis for task generation; The dynamic prompt word template incorporates a chain-thinking instruction, which requires the model to execute the following steps: Please follow these steps to think about and generate test cases: Analysis requirements: The analysis requirements include: understanding the functional, performance, and security requirements specified in the user's natural language query; Identification Interface: The identification interface includes: clearly defining all input and output variables of the system under test, their data types, and value ranges; Define test scenarios: The definition of test scenarios includes designing various test scenarios, including normal, boundary, and abnormal scenarios, based on requirements and interfaces; Write specific test cases for each test scenario; The dynamic prompt word template includes output format instructions; these instructions require the model to generate test case files with the .sss extension, indicating that: SSM::set <var> <val>< / val> < / var> SSM::cycle[ <integer> ]< / integer> Here, var refers to the name of the variable to be assigned, val refers to the specific value assigned to the var variable, and cycle refers to the number of cycles in which this abstract test case is run; Negative constraints and quality requirements are imposed on the generation of test cases; these include: it is strictly forbidden to fabricate or invent any requirements, interfaces or functions not mentioned in the provision context; the generated test cases must be repeatable; all declared exceptions and boundary conditions must be covered; and each test case must be able to be explicitly traced back to at least one requirement ID.

9. The method for automatically generating SCADE test cases based on a large language model according to claim 2, characterized in that: In step 6, the process of converting test cases into SCADE test sequences, driving the simulator to execute them, evaluating the execution results, and triggering confirmation output or feedback optimization in step 7 based on the evaluation results specifically includes: Convert test cases into SCADE test sequences to drive the simulator to execute them; The pass rate is calculated based on the execution results; the pass rate refers to the percentage of test cases that are successfully loaded and fully executed by the SCADE environment out of the total number of test cases. Test cases that cannot be loaded or executed will be immediately marked and categorized by the system. The execution results are used to statistically analyze whether simulation anomalies were triggered during execution, in order to verify whether the expected dynamic behavior of the model in the test cases matches the actual behavior. Calculate the requirement coverage rate based on the execution results; the requirement coverage rate refers to the percentage of requirement items verified by the test cases that have passed the tests out of the total number of all relevant requirement items; Calculate branch coverage and MC / DC coverage based on the execution results; branch coverage refers to the percentage of covered model logic branches out of all branches; The calculated indicators are compared with the preset quality thresholds. If all indicators meet the standards, the generated test cases are confirmed as valid outputs after the evaluation process is completed. If any indicator fails to meet the standards, step 7 is executed for feedback optimization.

10. The method for automatically generating SCADE test cases based on a large language model according to claim 9, characterized in that: The process of taking the substandard evaluation results and their root causes as input, passing them to the prompt word optimizer, driving it to adjust the RAG search strategy or prompt word template parameters, and restarting the generation process starting from step 5 until the output results pass the evaluation, specifically includes: By analyzing the prompts and RAG search results used when generating this use case, we traced back to the text blocks and metadata in the vector database that provided the source context to locate the source of the erroneous knowledge. If the problem is determined to be due to a lack of knowledge, the missing correct knowledge, including the latest design documents and revised requirements specifications, will be regenerated into vector data records and added to the vector database. If the problem is determined to be due to knowledge errors or obsolescence, the corresponding text block records in the vector database are marked, including adding the deprecated tag to their metadata, reducing the weight of the record during the retrieval process, or completely removing it from the database. Continuously monitor and record the evaluation metrics of the test cases generated under each prompt word template version, and establish a quantitative mapping relationship between prompt word design and generation effect; To address specific performance shortcomings, different prompt word optimization strategies are deployed in parallel for A / B testing, and a new template version with better performance is selected based on the test data. Version control will be implemented for the optimized prompt word templates.

Images