Automatic driving test case generation system based on large language model

Abstract

The embodiment of the application discloses an automatic driving test case generation system based on a large language model, the generation system comprises a multi-modal data preprocessing module, a large model generation module and a retrieval enhancement generation module; test cases are gradually generated in combination with functional features, the test coverage depth of complex control functions and the engineering execution feasibility are significantly improved; and whether the current test item covers all functional paths, state transitions or abnormal scenes is actively judged, gaps are completed, and the coverage requirements of the automatic driving system on redundant paths and failure protection are met; in addition, the subjective fluctuation problem of the general model on test semantic understanding is solved, a complete closed loop of structured scoring + automatic revision + explanation output is realized, and the standardization degree of test cases and the platform integration are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving test case generation, and in particular to an autonomous driving test case generation system based on a large language model. Background Technology

[0002] The extreme complexity of autonomous driving systems necessitates testing that covers a vast array of driving scenarios. However, current test case generation methods commonly used by OEMs still face significant challenges when dealing with complex driving scenarios and unknown environmental conditions.

[0003] 1. When dealing with complex driving scenarios and unknown environmental conditions, existing methods suffer from problems such as low scenario coverage and low generation efficiency;

[0004] 2. Although some test cases can be generated with the help of large models, their output quality generally does not meet the application standards and still requires manual review and correction, resulting in high labor costs and difficulty in improving coverage.

[0005] 3. As autonomous driving functions continue to expand and be optimized, test scenarios are becoming more diversified and refined, which in turn leads to a continuous increase in the number of test cases;

[0006] 4. Traditional manually designed test cases focus on a single function and cannot be quickly adapted to multi-functional collaborative scenarios, resulting in low testing efficiency and incomplete scenario coverage, which seriously restricts test coverage and result accuracy.

[0007] Against this backdrop, the value of automated test case generation technology becomes apparent—it reduces testing costs and resource consumption by minimizing manual intervention. Therefore, achieving high-precision and high-efficiency automated test case generation in the field of autonomous driving has become a core technological bottleneck that the industry urgently needs to overcome. Summary of the Invention

[0008] To address the problems in the background art, this invention proposes an autonomous driving test case generation system based on a large language model. The generation system includes a multimodal data preprocessing module, a large model generation module, and a retrieval enhancement generation module.

[0009] The multimodal data preprocessing module is used to receive the original functional requirement document and the user-input functional requirement to be tested, and to perform preprocessing on the original functional requirement document by extracting, transforming and replacing the functional requirement to be tested to obtain the functional requirement text.

[0010] The large model generation module is used to receive the initial prompt words input by the user and the text of the function requirement to be tested, and to call the search enhancement generation module based on the initial prompt words and the text of the function requirement to be tested;

[0011] The retrieval enhancement generation module is used to vectorize the text of the functional requirement to be tested, and to perform vector retrieval on the vectorized text of the functional requirement to be tested based on the initial prompt words to obtain the corresponding text fragments, and input them together with the initial prompt words into the large model generation module;

[0012] The large model generation module is used to divide the test case generation task into multiple stages. Based on the initial prompt words and their related text fragments, and the text of the functional requirements to be tested, the module constructs corresponding prompt word information for each stage in sequence. Based on the text of the functional requirements to be tested and the prompt word information, the module calls the retrieval enhancement module and guides the preset large language model to output a set of candidate test cases.

[0013] Furthermore, the generation system also includes a feedback optimization module;

[0014] The feedback optimization module is used to receive the candidate test case set and the text of the functional requirements to be tested, and to evaluate and correct each candidate test case in the candidate test case set based on the fine-tuned large language model and the general large language model, so as to generate the target test case set.

[0015] 3. The generation system according to claim 2, characterized in that the fine-tuning process of the large language model includes:

[0016] Based on the formatted preset dataset, the pre-trained large language model is fine-tuned using LoRA technology and Adam optimizer until the pre-trained large language model converges, resulting in the fine-tuned large language model.

[0017] The loss function used for training is:

[0018]

[0019] Where N is the sequence length, It is the real word at the i-th position in the sequence. This refers to the probability predicted by the pre-trained large language model of the true word at the i-th position given the first i-1 words. 4. The generation system according to claim 1, characterized in that the preprocessing steps of the multimodal data preprocessing module include:

[0020] Receive the original functional requirements document and the user-input functional requirements to be tested, and extract the relevant parts from the original functional requirements document based on the functional requirements to be tested to obtain the functional requirements document to be tested;

[0021] The visual elements in the functional requirements document to be tested are identified by a multimodal large model and converted into natural language descriptive text.

[0022] The visual elements are replaced with the corresponding natural language description text, and combined with the text in the functional requirement document to be tested to form the functional requirement text to be tested.

[0023] 5. The generation system according to claim 4, wherein the visual element includes a simple table, the multimodal data preprocessing module further includes a conversion process for the simple table, the simple table includes a table title, a first row as a table header and other rows of data rows excluding the first row, and the natural language description text includes simple table description text;

[0024] Extract the table title, the table header, and the data rows. Based on the table header and the table title, concatenate the contents of the data rows according to a preset concatenation rule to form a corresponding simple table description text.

[0025] 6. The generation system according to claim 1, characterized in that the retrieval enhancement generation module vectorizes and performs vector retrieval on the text of the functional requirement to be tested, including:

[0026] The text of the functional requirement to be tested is segmented into several text fragments, and the several text fragments are converted into corresponding vectors using a pre-set text embedding model. The resulting vectors are then stored in a pre-set vector database.

[0027] Calculate the similarity between the vectorized query question and each vector in the vector database to obtain multiple similarity results;

[0028] The text segments corresponding to the multiple similarity scores are sorted according to their scores, and the text segment most relevant to the query question is selected and input into the large model generation module.

[0029] 7. The generation system according to claim 1, characterized in that the prompt word information is mapped to a query question, the prompt word information is any one of first prompt word information, second prompt word information, and supplementary functional prompt word information, and the large model generation module includes a large language model for completing the task of generating a candidate test case set, including:

[0030] The first prompt word information, constructed based on the initial prompt word and its related text fragments, is input into a preset large language model, so that the large language model outputs multiple test sub-functions and their corresponding test strategies based on the first prompt word information and the text of the required function to be tested.

[0031] Traverse the test sub-functions and their test strategies, and supplement the corresponding function prompt words to the preset prompt word template according to the function type of the test sub-function. The prompt word template after supplementing the function prompt words will guide the large language model to call the retrieval enhancement generation module to retrieve related text fragments from the text of the function requirement to be tested, and guide the large language model to generate the corresponding initial test item set.

[0032] Based on the supplementary information of the supplementary prompt words, a second prompt word information is constructed using the prompt word template. Based on the second prompt word information, a preset prompt word template and its related text fragments and an initial test item set are filled in to guide the large language model to generate a candidate test case set.

[0033] 8. The generation system according to claim 7, wherein each of the function types corresponds to different function prompt words, and the function type is any one of process type, status type, sequence type, security strategy type or description type;

[0034] The fine-tuned sub-function type selection model identifies the function type to which the test sub-function belongs, and fills the corresponding function prompt word supplementary information into the corresponding position of the preset prompt word template according to the function type.

[0035] 9. The generation system according to claim 8, characterized in that training the fine-tuned sub-function type selection model includes:

[0036] Based on the formatted preset dataset, the pre-trained large language model is fine-tuned using LoRA technology and Adam optimizer until it converges, resulting in the fine-tuned large language model.

[0037] 10. The generation system according to claim 8, wherein the large model generation module further includes a preset coverage analysis mechanism based on the functional type;

[0038] Based on the initial test item set, the text of the functional requirements to be tested, and the preset coverage analysis mechanism, it is determined whether the current test item set has covered the target functional space;

[0039] If yes, then the initial test item set is used as the candidate test item set; otherwise, the large language model is invoked to automatically generate corresponding test items based on the missing paths in the initial test item set, thus obtaining the candidate test item set. The embodiments of this application have the following beneficial effects:

[0040] This application discloses an autonomous driving test case generation system based on a large language model. The system proposed in this invention not only generates test cases step-by-step by combining functional features, significantly improving the test coverage depth and engineering execution feasibility of complex control functions; it also proactively determines whether the current test item covers all functional paths, state transitions, or abnormal scenarios, filling gaps and meeting the coverage requirements of autonomous driving systems for redundant paths and failure protection; furthermore, it solves the problem of subjective fluctuations in the semantic understanding of general models, realizing a complete closed loop of structured scoring + automatic revision + interpretation output, greatly improving the standardization of test cases and the integration of the platform. Attached Figure Description

[0041] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and therefore should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.

[0042] Figure 1 This paper shows a schematic diagram of the structure of the autonomous driving test case generation system based on a large language model proposed in an embodiment of this application;

[0043] Figure 2 This paper illustrates a flowchart of the multimodal data preprocessing process in the generation system proposed in an embodiment of this application.

[0044] Figure 3 This paper illustrates a flowchart of the multimodal data preprocessing process in the generation system proposed in an embodiment of this application.

[0045] Figure 4 This paper shows a schematic diagram of the directory structure of the files listed in the generation system proposed in an embodiment of this application;

[0046] Figure 5 This illustration shows a simplified table processing diagram of the data preprocessing module in the generation system proposed in an embodiment of this application;

[0047] Figure 6 This paper illustrates a flowchart of the usage process of the retrieval enhancement generation module in the generation system proposed in an embodiment of this application;

[0048] Figure 7 A flowchart illustrating the large model generation module of the generation system proposed in this application is shown.

[0049] Figure 8 This is a schematic diagram of the prompt word template for generating the sub-functions to be tested and the test strategy in the large model generation module of the autonomous driving test case generation system based on a large language model provided in an embodiment of the present invention.

[0050] Figure 9 This is a schematic diagram of the prompt word template for generating test items in the large model generation module of the autonomous driving test case generation system based on a large language model provided in an embodiment of the present invention.

[0051] Figure 10 This is a schematic diagram of the prompt word template for generating test conditions, test steps, and expected results in the large model generation module of the autonomous driving test case generation system based on a large language model provided in an embodiment of the present invention.

[0052] Figure 11 This is a schematic diagram of the feedback optimization module structure of the autonomous driving test case generation system based on a large language model provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] In existing related technologies, test case generation mainly relies on rule-based generation and model-based generation. Rule-based generation methods depend on manually set fixed logic and templates, lacking flexibility and struggling to cope with the diverse structures and semantic complexities of requirements documents. While model-based generation methods incorporate statistical analysis and mathematical modeling, they often lack the ability to understand natural language and struggle to handle multimodal inputs such as charts, resulting in insufficient semantic coverage and limited expressiveness. Consequently, they suffer from low generation accuracy and high levels of human intervention.

[0055] To address the aforementioned issues, this invention provides an autonomous driving test case generation system based on a large language model. This system is particularly suitable for testing complex functional modules in the autonomous driving field. Through multimodal data preprocessing, enhanced retrieval generation, iterative design of prompt words, and feedback optimization mechanisms, this system supports the automatic generation of structurally complete, logically consistent, and highly covered test cases from different types of functional requirement documents. This not only significantly reduces the manual workload but also improves the efficiency and quality of test design.

[0056] To more clearly demonstrate the implementation steps and advantages of this invention, the specific implementation methods are described below with reference to the illustrations.

[0057] This application proposes an autonomous driving test case generation system based on a large language model, such as... Figure 1As shown, this autonomous driving test case generation system based on a large language model includes a multimodal data preprocessing module, a large model generation module, and a retrieval enhancement generation module. These modules work collaboratively to automate the generation process from raw functional requirement documents to structured, high-quality test cases, and from multimodal document parsing to test case optimization, thus comprehensively improving the semantic fit, structural integrity, and engineering usability of test cases. The specific details of the generation system are as follows:

[0058] The multimodal data preprocessing module receives the original functional requirements document and the user-input functional requirements to be tested. Based on the functional requirements to be tested, it extracts, transforms, and replaces the relevant parts from the original functional requirements document to form the functional requirements text to be tested.

[0059] The multimodal data preprocessing module receives the input original functional requirements document and extracts the information needed to generate test cases based on the functional requirements to be tested. The original functional requirements document for autonomous driving systems typically contains multimodal data, including text, tables, images, and other information formats. Tables can be simple or complex. This multimodal data preprocessing module uses text processing tools and a multimodal large language model to uniformly identify and parse the text, tables, images, and other modal content in the original functional requirements document, automatically converting them into natural language descriptions and saving them as the text of the functional requirements to be tested. This processing method solves the problem of poor adaptability of traditional methods when parsing documents with mixed text and images, enhancing the generation system's ability to handle multiple input formats. For example, a multimodal large model can be used. This multimodal data preprocessing module supports structured parsing of functional specifications in editable PDF format. Introducing the multimodal large model allows for natural language conversion of chart and visual information, i.e., semantic conversion, achieving a unified transformation of tables and images into natural language. This ensures the semantic integrity and consistency of the original functional requirement document content, significantly improving the system's ability to process automotive-grade engineering documents. The multimodal large model can be Qwen2.5-VL, such as Qwen2.5-VL-3B. Qwen2.5-VL possesses multimodal understanding capabilities, accurately parsing objects, text, charts, and other elements in images, and supporting the understanding and event localization of long video content exceeding one hour.

[0060] like Figure 2 As shown, the multimodal data preprocessing module, through a series of steps including file format conversion, functional requirement document analysis, and image-text conversion, accurately extracts information from the original functional requirement document and performs preprocessing such as conversion and replacement to obtain the functional requirements to be tested, providing high-quality input data for subsequent modules. This multimodal data preprocessing module specifically includes:

[0061] Obtain the original functional requirements document. Each original functional requirements document contains at least one functional requirement to be tested. Read the complete original functional requirements document and decompose at least one functional requirement to be tested into corresponding functional requirements documents. The original functional requirements document can be an editable PDF file. The multimodal data preprocessing module will use a regular expression matching algorithm to ensure that the content of the original functional requirements document is accurately extracted, providing a foundation for subsequent data processing.

[0062] The extracted complete content of the functional requirements document to be tested includes: receiving user-input functional requirements and mapping them to title instructions. Different title instructions correspond to different title lines. Each original functional requirements document may include at least one level of headings, such as level 1, level 2, and level 3 headings. The starting position of the extracted content is determined according to the title line corresponding to the title instruction (i.e., the current title line). The text from the current title line to the next sibling title line is extracted. If there are no subsequent sibling title lines, the content from the current title line to the previous title line is extracted. If there are no subsequent title lines, only the text from the current title line to the end of the original functional requirements document is extracted to ensure that the complete content of the current title line is captured. The extracted information fragments will retain the complete content of the original text and maintain the original format, i.e., maintain the Markdown format.

[0063] Exemplary, such as Figure 3 As shown, if the original functional requirements document is "External Lighting Control," which contains multiple functional requirements to be tested, such as "Position Lights," "Daytime Running Lights," and "Reversing Function," the process begins by converting the editable PDF version of "External Lighting Control" into Markdown format. Then, regular expressions are used to match the heading lines of this document, extracting the hierarchical table of contents. For example, the "External Lighting Control" document contains four levels of headings; "5.2.3.1 Position Lights" is the content of a level 4 heading. Users can input the functional requirements to be tested in the interactive interface. In other words, they can select the corresponding heading line of the functional requirement to be tested from the drop-down menu in the interactive interface, such as selecting the heading line of "5.2.3.1 Position Lights." The corresponding content of that heading line is then extracted as a single functional requirement to be tested within the original functional requirements document.

[0064] The process handles multimodal data in the functional requirements document under test. It automatically identifies and converts visual elements such as tables and images in the document into natural language descriptions using a multimodal large-scale model. This addresses issues of mixed text and image content and unparseable information, ensuring that the content of the functional requirements document is transformed into a unified natural language description for subsequent search enhancement generation steps. For example, the table and image content after multimodal large-scale model transformation might look like this: Figure 4As shown, the conversion of the position light input signals in Table 4.5.3 involves converting each item in the table (the table header and its corresponding data). This includes describing the table's description, input / output, type, effective value (when on), and wet current, resulting in corresponding natural language description text. The converted natural language description text is then used to replace the corresponding images and tables in the functional requirements file to be tested, resulting in the optimized functional requirements file, i.e., the functional requirements text.

[0065] In addition, the multimodal data preprocessing module also includes processing simple tables (simple tables). Simple tables contain only a table title, a first row as the header, and the remaining rows as data rows. The natural language description text includes the simple table description text. The processing for simple tables includes: converting the Markdown format functional requirement file to be tested into an HTML structure; using Beautiful Soup to parse the HTML (Hypertext Markup Language) structure; Beautiful Soup is a library that can easily extract information from web pages, sits on top of HTML or XML parsers, and provides Python idioms for iterating, searching, and modifying parse trees; using Beautiful Soup to extract the table title, header, and data rows of the simple table; clarifying the theme and purpose of the table content based on the table title; and concatenating the data row content according to the theme and purpose of the header and table content to obtain the simple table description text in natural language, combined according to pre-set concatenation rules. The simple table conversion is as follows: Figure 5 As shown, replacing the corresponding simple table with the combined simple table description text is faster than using a multimodal large model. When all tables are in simple table form, it is faster than the multimodal large model, reducing the processing time by more than 1 hour per 100 tables on average.

[0066] The multimodal data preprocessing module ultimately converts richly illustrated documents into text describing them in pure natural language. This method transforms visual elements such as tables and images into a unified natural language representation while preserving important information from the tables and images, making it more suitable for large language models to comprehensively understand the content of functional requirements documents.

[0067] The large model generation module receives the text of the functional requirements to be tested and the query questions to be queried from the multimodal data preprocessing module. The query questions include initial prompts input by the user, used to generate a set of candidate test cases. Specifically, the retrieval enhancement generation tool in the retrieval enhancement generation module is provided as a callable function to the large language model in the large model generation module. The query questions input to the large model generation module require the large language model to first call the retrieval enhancement generation tool to retrieve relevant text fragments from the text of the functional requirements to be tested each time it generates an answer, and then input the relevant text fragments and the query questions together into the large model generation module to generate an answer. For example... Figure 6 As shown in the flowchart of the retrieval enhancement generation module, the retrieval enhancement generation tool segments the text of the functional requirement to be tested into several text fragments. It then uses a pre-set text embedding model to convert these text fragments into vectors and stores the resulting vectors in a pre-set vector database. Through vectorization and vector retrieval of the text of the functional requirement to be tested, the retrieval enhancement generation module facilitates the accurate extraction of textual information related to the functional requirement, providing high-quality input data for the subsequent large-scale model generation module, as well as more accurate and professional semantic background information. This improves the relevance and rationality of the generated results, thereby enhancing the accuracy and semantic alignment of the generated test cases.

[0068] The retrieval enhancement generation tool calculates the similarity between the vectorized query question and each vector in the vector database. It then sorts the text fragments corresponding to each similarity vector based on their similarity scores, selecting the text fragment with the highest similarity score—that is, the text fragment most relevant to the query question—as contextual supplementation. This text fragment, along with the query question, is input into the large model generation module to support subsequent generation processes. The retrieval enhancement generation module pre-builds a vector database based on the text of the functional requirements to be tested, which improves the quality of the context input during the large model generation stage. The text embedding model and the pre-built vector database are well-known to those skilled in the art and will not be elaborated upon here. The query question can include initial prompts and prompt information.

[0069] The large-scale model generation module employs an iterative design mechanism based on prompt words, dividing the candidate test case generation task into multiple stages and dynamically guiding the large language model to complete each stage. Specifically, the test case generation task can be divided into three stages: test sub-function and test strategy generation stage, test item generation stage, and test step and expected result generation stage. First, the large language model is guided to generate test sub-functions and their corresponding test strategies based on the text of the functional requirements to be tested. Then, it iterates through each test sub-function and its test strategy combination, further retrieving relevant semantic fragments from the text of the functional requirements to be tested, i.e., the retrieved relevant text fragments are added to the preset prompt word template to guide the large language model to generate corresponding test items. Finally, through multiple rounds of prompt word completion and reasoning guidance, test conditions, test steps, and expected results are gradually generated, forming a set of test cases that are structurally complete, semantically clear, and conform to the characteristics of autonomous driving tasks. In this process, corresponding prompt words are constructed at each stage and combined with relevant text fragments provided by the retrieval enhancement generation module. These are then input into a pre-set large language model, such as Qwen2.5-14B, to achieve phased and structured generation of test cases, ensuring logical coherence and semantic consistency throughout the generation process. Pre-built prompt word templates are used to fill in various prompt words in subsequent stages. These prompt words include, but are not limited to, the first prompt word information, relevant text fragments retrieved from the test requirement text, supplementary prompt word information, and the second prompt word information. The prompt word information constructed at each stage includes task background prompts, output guidance prompts, and reasoning example prompts. Specifically, task background prompts explain the current task objective and inform the large language model of the large model generation module of the external tools that can be called (such as retrieval enhancement generation tools); output guidance prompts clarify the reasoning logic and generation structure, introducing a chain-of-thought step-by-step reasoning strategy; and reasoning example prompts provide standardized output format examples to guide the large language model to align its generation style and logic. The prompt words, combined with dynamic contextual hint chains, are input into the locally deployed large language model, enabling full control over the structured generation of test cases. Inference example hints can be provided by the user or pre-set.

[0070] In addition, to further adapt to the differentiated generation needs of different structural functions in autonomous driving, the large model generation module also includes a sub-function type selection mechanism. This mechanism automatically identifies the function type of the sub-function to be tested by using a sub-function type selection model fine-tuned based on instruction fine-tuning and parameter efficient fine-tuning technology LoRA (Low-Rank Adaptation) according to the semantic structural features of the sub-function to be tested. It then matches the corresponding function prompt words and fills them into the corresponding positions in the preset prompt word template, thereby guiding the large language model to generate more targeted and structurally adaptable test content to improve the semantic adaptability and generation quality of the test items.

[0071] The functional types of the sub-functions under test include, but are not limited to, any one of the following: process-oriented, state-oriented, sequential, security policy-oriented, or descriptive. Process-oriented functions: path coverage-based prompts are preferred; state-oriented functions: state transition-driven prompts are used; sequential functions: time-driven and event chain-based prompts are used; security policy-oriented functions: protection boundary and fault injection-related prompts are introduced; descriptive functions: condition-action-verification structure prompts are used to guide the generation of test verification items based on functional responses.

[0072] Specifically, the semantic information of the sub-function is input into the fine-tuned sub-function type selection model. After automatically identifying the function type to which the sub-function to be tested belongs, the model matches the corresponding supplementary information of the function prompt words according to its function type. The supplementary information of each function prompt word can be:

[0073] The prompt for a workflow-type function is: "Based on the sub-function description, please cover all normal paths, abnormal paths, and boundary paths to generate test items. The output test items should cover each key branch and decision condition."

[0074] Status-based function prompt: "Based on the sub-function description, analyze the current status list and event triggering conditions, and generate test items for each status and its legal or illegal transition paths."

[0075] The prompt for a sequence-based function is: "Based on the sub-function description, extract the key time sequence and event chain to generate test items that conform to the sequence logic, covering the system response behavior before and after the event is triggered."

[0076] Safety strategy-type function prompt: "Based on the sub-function description, please generate boundary value test and fault injection scenario items for this function to verify whether the system protection mechanism is effective, focusing on the input range, fault signals, system alarms and protection trigger conditions."

[0077] Description-based function prompt: "Based on the sub-function description, please generate test items that include prerequisites, trigger actions, and function response verification."

[0078] The specific training process of the sub-function type selection model is as follows:

[0079] 1. Dataset

[0080] The training data for the sub-function type selection model can be selected from functional requirement documents of real autonomous driving systems, covering various sub-function types. The data scale includes thousands of sub-function description texts with clear function type labels. Each training sample can adopt a JSON format structure. For example, the natural language text description of a sub-function, i.e., "After the vehicle starts, detect the engine status to ensure the engine runs normally," outputs the corresponding one-hot encoding. The corresponding function type label is determined based on the one-hot encoding. Different one-hot encodings correspond to different function type labels. One-hot encodings can be 10000, 01000, etc. The function type label set includes process type, status type, time series type, safety strategy type, and descriptive type. The above dataset can be divided into 80% training set, 10% validation set, and 10% test set, or it can be set according to the actual situation for training and evaluating model performance.

[0081] Before training the model, a series of preprocessing operations are required on the dataset. First, the training data (dataset) is loaded from a JSON file. This training data contains natural language text descriptions of sub-functions and corresponding function type labels. Next, the category labels are mapped to integer indices, and the integer indices are converted to One-Hot encoding. Then, the AutoTokenizer, the tokenizer of the pre-trained model, encodes the sub-function description text into a format that the sub-function type selection model can process. AutoTokenizer is a core utility class in the Hugging Face Transformers library used to automatically load pre-trained model tokenizers. It can dynamically match the corresponding tokenizer implementation based on the model name and supports multiple mainstream model architectures such as BERT, GPT, and LLaMA.

[0082] 2. Training process

[0083] The sub-function type selection model combines instruction fine-tuning and efficient parameter fine-tuning training methods. First, it uses the pre-trained large language model Qwen2.5-14B, whose input is sub-function description text and output is the corresponding function type. Then, LoRA technology is employed to perform low-rank adjustment on the q_proj, k_proj, and v_proj weight matrices in the self-attention layer of the sub-function type selection model, while freezing the remaining parameters. This significantly reduces training resource consumption, improves training efficiency, and prevents overfitting. During training, the model continuously updates its parameters by matching the input sub-function semantic information with the target type label, learning the structural classification rules of different semantic structures, and improving its intelligent matching ability for prompt word templates.

[0084] 3. Loss Function

[0085] Training uses the multi-class cross-entropy loss function, defined as:

[0086]

[0087] In the formula, One-hot encoding of the real label. The loss function is used to predict probabilities for the model, where N is the number of classes. This loss function can effectively drive the model to improve classification accuracy.

[0088] 4. Convergence Judgment

[0089] Model training convergence is achieved through validation set cross-entropy loss and accuracy monitoring. Convergence is determined when the validation loss shows no significant decrease over multiple consecutive epochs and the validation accuracy reaches or exceeds a preset first probability (e.g., 95%). Multiple independent training experiments show stable metrics, demonstrating that the sub-function type selection model exhibits stable training and good generalization performance.

[0090] By adopting the above training mechanism and sub-function type selection model, the semantic adaptability and quality of the large language model in generating test cases for various structured sub-functions in the field of autonomous driving can be significantly improved, ensuring that the test cases have stronger task relevance, logical integrity and engineering usability.

[0091] The large model generation module will guide the large language model to retrieve relevant content from the text of the functional requirements to be tested based on the initial input prompts and generate corresponding test items.

[0092] Figure 7 The flowchart for the large model generation module shows that, in its implementation, the test case generation task is divided into three stages. Each stage constructs corresponding prompt words, which are mapped to query questions. These prompt words can include first prompt words, second prompt words, and supplementary functional prompt words. The test case generation task is divided into three stages as follows:

[0093] Step 1: Test Sub-functions and Test Strategy Generation Phase

[0094] The system receives initial prompts and the text of the function requirement to be tested from the user. The initial prompts represent the query question. After receiving the initial prompts, the large language model calls the retrieval enhancement generation module. This module retrieves the most relevant text fragments from the function requirement text as context supplements to the initial prompts. The first prompt information, constructed from the relevant text fragments and the initial prompts, is then input into the large language model. The constructed first prompt information is as follows: Figure 8As shown. The first prompt word will guide the large language model to understand the current task objective, such as: "Your objective is to read the functional requirements document, deduce the key test points layer by layer, structure them into independent sub-functional items, and design test strategies for each sub-functional item to support efficient test case writing"; the large language model will output multiple test sub-functionalities based on the first prompt word and the text of the functional requirement to be tested, such as "turn on the position light", and the corresponding test strategies, such as "design a positive test strategy for turning on the position light".

[0095] Step 2: Test Item Generation Phase

[0096] Iterate through the above output test sub-functions and their test strategies, select the model to identify the function type of the test sub-function based on the sub-function type, and different function types correspond to different function prompt word supplementary information. The function prompt word supplementary information is mapped to prompt word information, and the corresponding function prompt word supplementary information is added to the preset prompt word template according to its function type. For example, as shown Figure 9 As shown, the function type of the test sub-function is identified as descriptive. The descriptive function prompt, "Please generate test items containing preconditions, trigger actions, and function response verification based on the sub-function description," will be filled into the {function_strategy} of the preset prompt template. The supplementary information of the function prompt details the tools that the large language model can call. The prompt template after supplementing the function prompt information will guide the large language model to call the retrieval enhancement generation module to retrieve relevant content from the text of the function requirement to be tested, and guide the large language model to generate the corresponding initial test items, such as: "When the system is in an unarmed state, check whether the position light is lit normally after the position light switch is turned on." At least one initial test item is combined to obtain the initial test item set.

[0097] After generating multiple test items, i.e., generating the initial test item set, a coverage analysis mechanism will be executed: combining the requirement text of the function to be tested and the current initial test item set, the large language model will determine whether the current initial test item set has covered the target functional space. The criteria for this coverage analysis mechanism will also be differentiated according to the functional type of the tested sub-functions, for example:

[0098] For process-oriented functions, test items are required to cover all control branches;

[0099] For state-based functions, test items must cover all state transition paths;

[0100] For time-series functions, test items are required to cover key time chains and event combinations;

[0101] For security policy-related functions, the test items must cover abnormal scenarios and fault inputs;

[0102] For descriptive functions, check whether the test items omit the trigger verification of core functional behaviors.

[0103] If yes, the current initial test item set will be used as the candidate test item set; otherwise, if the large language model determines that there is a coverage gap in the initial test item set, it will call the large language model to automatically generate corresponding test items based on the missing paths, thereby forming a test coverage set with a closed structure, that is, generating a candidate test item set.

[0104] Step 3: Test Procedures, Conditions, and Expected Results Generation Stage

[0105] Based on the supplementary prompt information provided above, a second prompt information is further constructed using the prompt template. This second prompt information is then filled into the preset prompt template. The second prompt information is as follows: Figure 10 The second prompt word information will guide the large language model to combine the aforementioned test sub-functions, test strategies, and test items to generate a complete test process. In other words, the above-mentioned candidate test item set, relevant text fragments, and second prompt word information are input into the large language model. The complete test process output by the large language model includes test conditions, test steps, and expected results, that is, it outputs corresponding candidate test cases. Multiple candidate test cases form a candidate test case set. The specific output of the complete test process includes:

[0106] Test conditions, such as "the system is in an unarmed state; the position light switch is on";

[0107] Test procedures, such as "1. Check if the system is in an unarmed state; 2. Turn the position light switch to the ON position";

[0108] Expected result: "The value of BCM_PositionLightCmd in the CAN message should be 0x1, and the position light should light up normally."

[0109] In the above-mentioned multi-stage test case generation process, the large language model dynamically integrates the preceding output content, that is, dynamically integrates the preceding output test sub-functions with their test strategies and test items to form a complete test context, that is, a complete test process, which can effectively improve the logical coherence and coverage completeness of the generated candidate test cases.

[0110] The feedback optimization module is used to automatically assess and correct the quality of generated test cases. This module receives the text of the functional requirements to be tested and a set of candidate test cases as input. Based on a fine-tuned local large language model (e.g., Qwen2.5-14B), each candidate test case is given a semantic quality score. Evaluation dimensions include, but are not limited to: clarity, logical consistency, execution feasibility, format standardization, and functional alignment. The large language model outputs a structured score result and evaluation explanation for each candidate test case. For test cases that do not meet the score or have structural defects, the large language model will automatically fill in missing information, fix logical conflicts, and unify the expression style to generate revised test cases (i.e., revised versions) based on the evaluation dimensions and the reasons for the score explanation. High-quality test cases optimized by feedback will be exported and saved in a structured format (e.g., CSV file), achieving system-level closed-loop optimization and directly connecting to automated testing platforms.

[0111] Figure 11 This is a flowchart illustrating the feedback optimization module. After candidate test cases are generated, this module automatically assesses and corrects the generated test cases. The module receives the text of the functional requirements to be tested and candidate test cases as input. Based on a pre-set, fine-tuned local large language model (i.e., a fine-tuned large language model), it scores the semantic quality of each candidate test case, independently evaluates the semantic quality and structural consistency of each test case, and improves its engineering usability through an automatic revision mechanism, thereby generating a target test case set. The scoring dimensions of the feedback optimization module include, but are not limited to, clarity, logical consistency, execution feasibility, format standardization, and functional alignment.

[0112] This feedback optimization module will employ a large language model fine-tuned based on test case corpora from the autonomous driving field as its core evaluation engine, replacing the application of general pre-trained large models or traditional rules in the evaluation process. In comparison, while general large language models possess language understanding capabilities, they suffer from inconsistent scoring dimensions in structured testing tasks, are susceptible to superficial language style interference leading to subjective fluctuations in scoring results, and cannot output structured evaluation opinions. Traditional rule-based methods cannot handle semantic-level issues, failing to assess logical consistency, execution integrity, or generate revision suggestions. In contrast, the fine-tuned large language model, trained on professional test data, offers the following advantages: it accurately identifies the causal relationships between test conditions, operational steps, and expected results; it adheres to unified and clear scoring dimensions, avoiding evaluation subjectivity and standard deviation; and it outputs structured scoring results and explanatory reasoning, exhibiting higher consistency and fairness.

[0113] Specifically, the feedback optimization module includes the following two stages:

[0114] Phase 1: Test Case Quality Assessment

[0115] The pre-set fine-tuned large language model is invoked to score the semantic quality of each candidate test case. The evaluation dimensions are as follows: "Clarity: Is the language expression concise and clear? Logical consistency: Do the conditions, steps and expected results correspond reasonably? Execution feasibility: Does it have complete premises and operable steps? Format standardization: Does it conform to the standard test case format? Functional alignment: Does it accurately reflect the requirements of the function to be tested? The model outputs structured scoring results and evaluation explanations for each test case."

[0116] For example, “Sub-function: Position light turned on; Test strategy: Forward design for position light turning on; Test item: Check if the light is lit after turning on the position light switch; Test condition: Position light switch is on. Test steps: Turn the position light switch to the ON position. Expected result: Position light lights up. Scoring results: Clarity: 3; Logical consistency: 2; Execution feasibility: 2; Formatting standardization: 3; Functional alignment: 4. Evaluation conclusion: This test case does not describe the initial state of the system, which may pose a risk of accidental test path activation; the test steps are too simplified and do not include prerequisite checks; the expected result lacks signal verification indicators and is not expressed in a standardized manner.”

[0117] Phase Two: Test Case Revision and Optimization

[0118] For candidate test cases that fail to meet the scoring criteria or have structural defects, a revised version will be generated using a general large language model, i.e., a set of target test cases will be generated. The general large language model can be Qwen2.5-14B. The revision process may include: the general large language model reading the functional requirements document to be tested and supplementing missing test conditions or steps; fixing logical inconsistencies between steps and expectations; and adjusting the expression style to make it more standardized and consistent.

[0119] Example, revised version: "Sub-function: Position light on; Test strategy: Forward design for position light on; Test item: Turn on the position light in the system's unarmed state to verify whether the control command and physical response are consistent; Test conditions: The system is in an unarmed state; the position light switch is on; Test steps: Check whether the system is in an unarmed state; Turn the position light switch to the ON position; Expected result: The value of BCM_PositionLightCmd in the CAN message should be 0x1; The position light is lit normally, without any abnormal status or alarm."

[0120] Specifically, the feedback optimization module will use a large language model fine-tuned based on test case corpus in the autonomous driving field as the core evaluation engine, and combine instruction fine-tuning and efficient parameter fine-tuning techniques to achieve accurate candidate test case quality scores. The fine-tuning process of the large language model in the first stage of the feedback optimization module is as follows:

[0121] 1. Dataset

[0122] The dataset will be in JSON format, and each sample will consist of two parts:

[0123] Context: The complete text containing the functional requirements description and all test case elements (sub-functions, test strategies, test items, test conditions, test steps, expected results) and scoring instructions.

[0124] Response: Includes structured multi-dimensional scores (such as clarity, logical consistency, feasibility, formatting, and functional alignment, with a score range of 1-5 points) and corresponding natural language evaluation explanations.

[0125] Example, data format:

[0126] {

[0127] "context": [

[0128] {

[0129] "role": "user",

[0130] "content": "Functional Requirements: 1) Under unarmed conditions, the position light turns on when the switch is turned on; the position light turns off when the switch is turned off. Test Item: Under unarmed conditions, check whether the position light turns on normally when the switch is turned on. Please rate this test case (1-5 points) and evaluate it from the following 5 dimensions: clarity, logical consistency, execution feasibility, format standardization, and functional alignment."

[0131] }

[0132] ],

[0133] "response": {

[0134] "role": "assistant",

[0135] "content": "Dimensional Rating: Clarity: 5; Logical Consistency: 5; Execution Feasibility: 5; Formatting Standardization: 4; Functional Alignment: 5; Evaluation Conclusion: Covers the main functional logic, with clear steps and complete verification methods."

[0136] }

[0137] }

[0138] 2. Training process

[0139] Based on a pre-trained large language model (e.g., Qwen2.5-14B), and combined with LoRA (Low-Rank Adaptation) technology, only the parameters of key sub-modules in the pre-trained large language model are adjusted in low rank during training. Specifically, this includes the weight matrices of q_proj, k_proj, and v_proj in the self-attention layer. Except for the parameters of the key sub-modules, the other parameters of the pre-trained model remain frozen, thereby significantly reducing computational resource consumption, improving training efficiency, and enhancing the model's generalization ability.

[0140] During training, the pre-trained large language model takes formatted test case context and scoring task instructions as input, and outputs the probability distribution of multi-dimensional scores and corresponding evaluation explanation text. Training uses a mini-batch data approach, performing forward propagation to calculate the output sequentially, calculating the loss based on the actual labels, and then calculating the gradient through the backpropagation algorithm.

[0141] The parameter update process is performed using the Adam (Adaptive Moment Estimation) optimizer, with a learning rate set to 2×10⁻⁶. -4 Backpropagation only updates the gradients of the low-rank adjustment parameters of the LoRA module; the other weights of the main model do not participate in the update, ensuring the conservation of training resources and the stability of the training process. During training, the model is periodically evaluated on the validation set, including scoring accuracy and generated text quality metrics (such as BLEU, ROUGE), to monitor the training effect.

[0142] This approach effectively utilizes the powerful language understanding capabilities of pre-trained large language models, while ensuring efficient training and good generalization performance through fine-tuning of key parameters.

[0143] 3. Loss Function

[0144] For each input sequence, the pre-trained large language model outputs a predicted word probability distribution. The cross-entropy loss function calculates the difference between this predicted word probability distribution and the true distribution of the actual target words.

[0145]

[0146] Where N is the sequence length, It is the real word at the i-th position in the sequence. It is the probability that the model predicts the true word at the i-th position given the first i-1 words.

[0147] During training, the pre-trained large language model calculates the total loss through forward propagation, calculates the gradient through backpropagation, and updates the LoRA module parameters through the Adam optimizer. The main model parameters are kept frozen, which improves training efficiency and avoids overfitting.

[0148] 4. Convergence Judgment

[0149] Model training convergence is achieved through validation set cross-entropy loss and accuracy monitoring. For example, if there is no significant decrease in validation loss after 5 consecutive epochs and the validation accuracy reaches or exceeds a preset accuracy level (95%), training convergence is determined. Multiple independent training experiments show stable metrics, demonstrating model training stability and good generalization performance, resulting in a fine-tuned large language model.

[0150] The revised candidate test case structure is more standardized and semantically clear, possessing engineering executability and enhancing the depth of functional verification, thus meeting the requirements of an explanatory testing strategy. The final generated target test case set will meet preset standards in terms of semantic logic, format structure, and actual execution feasibility, and can be exported in CSV format for use by downstream testing platforms.

[0151] Through the aforementioned scoring and revision phases, the feedback optimization module enables automatic semantic evaluation, defect localization, and optimization of test cases. Combined with the unified scoring criteria and reasoning capabilities of the fine-tuned large language model, it significantly improves the performance of test cases in terms of logical integrity, language standardization, and engineering practicality, providing stable support for high-quality testing of autonomous driving systems.

[0152] In summary, the autonomous driving test case generation system provided by this invention, based on multimodal data preprocessing and large language model generation capabilities, combined with prompt word iterative design and feedback optimization mechanisms, can efficiently and accurately generate high-quality structured test cases from the original requirements document, greatly reducing the manual burden and improving the automation level and practicality of test generation.

[0153] The technical advantages and beneficial effects of this invention for autonomous driving systems are reflected in the following aspects:

[0154] (1) A phased, structured generation process ensures a logical closed loop for test cases.

[0155] By gradually generating test cases based on functional characteristics, the test coverage depth and engineering execution feasibility of complex control functions are significantly improved.

[0156] (2) Dynamic matching mechanism of prompt words based on semantic recognition

[0157] Test content is generated by differentiating structural types to ensure that the generated results are targeted and reliable in high-risk autonomous driving scenarios such as safety strategies and control timing.

[0158] (3) Closed-loop generation mechanism of integrated coverage analysis

[0159] The large language model will proactively determine whether the current test item covers all functional paths, state transitions, or abnormal scenarios, fill in the gaps, and meet the coverage requirements of the autonomous driving system for redundant paths and failure protection.

[0160] (4) Fine-tune model-driven scoring and revision to improve standard consistency.

[0161] It solves the problem of subjective fluctuations in the semantic understanding of general-purpose large language models, and realizes a complete closed loop of structured scoring + automatic revision + explanation output, which can significantly improve the standardization of test cases and the integration of the platform.

[0162] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An autonomous driving test case generation system based on a large language model, characterized in that, The generation system includes a multimodal data preprocessing module, a large model generation module, and a retrieval-enhanced generation module; The multimodal data preprocessing module is used to receive the original functional requirement document and the user-input functional requirement to be tested, and to perform preprocessing on the original functional requirement document by extracting, transforming and replacing the functional requirement to be tested to obtain the functional requirement text. The large model generation module is used to receive the initial prompt words input by the user and the text of the function requirement to be tested, and to call the search enhancement generation module based on the initial prompt words and the text of the function requirement to be tested; The retrieval enhancement generation module is used to vectorize the text of the functional requirement to be tested, and to perform vector retrieval on the vectorized text of the functional requirement to be tested based on the initial prompt words to obtain the corresponding text fragments, and input them together with the initial prompt words into the large model generation module; The large model generation module is used to divide the test case generation task into multiple stages. Based on the initial prompt words and their related text fragments, and the text of the functional requirements to be tested, the module constructs corresponding prompt word information for each stage in sequence. Based on the text of the functional requirements to be tested and the prompt word information, the module calls the retrieval enhancement module and guides the preset large language model to output a set of candidate test cases.

2. The generation system according to claim 1, characterized in that, The generation system also includes a feedback optimization module; The feedback optimization module is used to receive the candidate test case set and the text of the functional requirements to be tested, and to evaluate and correct each candidate test case in the candidate test case set based on the fine-tuned large language model and the general large language model, so as to generate the target test case set.

3. The generation system according to claim 2, characterized in that, The fine-tuning process of the large language model includes: Based on the formatted preset dataset, the pre-trained large language model is fine-tuned using LoRA technology and Adam optimizer until the pre-trained large language model converges, resulting in the fine-tuned large language model. The loss function used for training is: ， Where N is the sequence length, It is the real word at the i-th position in the sequence. It is the probability of the true word at the i-th position predicted by the pre-trained large language model, given the first i-1 words.

4. The generation system according to claim 1, characterized in that, The preprocessing steps of the multimodal data preprocessing module include: Receive the original functional requirements document and the user-input functional requirements to be tested, and extract the relevant parts from the original functional requirements document based on the functional requirements to be tested to obtain the functional requirements document to be tested; The visual elements in the functional requirements document to be tested are identified by a multimodal large model and converted into natural language descriptive text. The visual elements are replaced with the corresponding natural language description text, and combined with the text in the functional requirement document to be tested to form the functional requirement text to be tested.

5. The generation system according to claim 4, characterized in that, The visual elements include simple tables, and the multimodal data preprocessing module further includes transformation processing of the simple tables. The simple tables include table titles, a first row as the table header, and other rows of data rows except for the first row. The natural language description text includes simple table description text. Extract the table title, the table header, and the data rows. Based on the table header and the table title, concatenate the contents of the data rows according to a preset concatenation rule to form a corresponding simple table description text.

6. The generation system according to claim 1, characterized in that, The retrieval enhancement generation module vectorizes and performs vector retrieval on the text of the functional requirement to be tested, including: The text of the functional requirement to be tested is segmented into several text fragments, and the several text fragments are converted into corresponding vectors using a pre-set text embedding model. The resulting vectors are then stored in a pre-set vector database. Calculate the similarity between the vectorized query question and each vector in the vector database to obtain multiple similarity results; The text segments corresponding to the multiple similarity scores are sorted according to their scores, and the text segment most relevant to the query question is selected and input into the large model generation module.

7. The generation system according to claim 1, characterized in that, The prompt word information is mapped to a query question. The prompt word information is any one of the first prompt word information, the second prompt word information, and supplementary functional prompt word information. The large model generation module includes a large language model to complete the task of generating a candidate test case set, including: The first prompt word information, constructed based on the initial prompt word and its related text fragments, is input into a preset large language model, so that the large language model outputs multiple test sub-functions and their corresponding test strategies based on the first prompt word information and the text of the required function to be tested. Traverse the test sub-functions and their test strategies, and supplement the corresponding function prompt words to the preset prompt word template according to the function type of the test sub-function. The prompt word template after supplementing the function prompt words will guide the large language model to call the retrieval enhancement generation module to retrieve related text fragments from the text of the function requirement to be tested, and guide the large language model to generate the corresponding initial test item set. Based on the supplementary information of the supplementary prompt words, a second prompt word information is constructed using the prompt word template. Based on the second prompt word information, a preset prompt word template and its related text fragments and an initial test item set are filled in to guide the large language model to generate a candidate test case set.

8. The generation system according to claim 7, characterized in that, Each of the aforementioned function types corresponds to different supplementary information for function prompts. The function type can be any one of the following: process type, status type, sequence type, security policy type, or explanatory type. The fine-tuned sub-function type selection model identifies the function type to which the test sub-function belongs, and fills the corresponding function prompt word supplementary information into the corresponding position of the preset prompt word template according to the function type.

9. The generation system according to claim 8, characterized in that, Training the fine-tuned sub-function type selection model includes: Based on the formatted preset dataset, the pre-trained large language model is fine-tuned using LoRA technology and Adam optimizer until it converges, resulting in the fine-tuned large language model.

10. The generation system according to claim 8, characterized in that, The large model generation module also includes a preset coverage analysis mechanism based on the aforementioned function type; Based on the initial test item set, the text of the functional requirements to be tested, and the preset coverage analysis mechanism, it is determined whether the current test item set has covered the target functional space; If so, the initial set of test items will be used as the candidate set of test items; If not, the large language model will be invoked to automatically generate corresponding test items based on the missing paths in the initial test item set, thus obtaining the candidate test item set.

Images