Extraction method of key test scenarios for intelligent vehicles based on large language model

Abstract

The present invention addresses the technical problems in the process of extracting key test scenarios for smart cars, and provides a method for extracting key test scenarios for smart cars based on a large language model. The method relates to the technical field of smart car testing. The present invention utilizes a micro-parameter model to reasonably and accurately extract key test scenarios based on a small sample local private knowledge base. The method includes real vehicle data collection, obtaining scenario descriptions based on a large language model, extracting key test scenarios based on a small model, local deployment, and analysis of experimental results. Experimental results show that compared with an online LLM without a local private knowledge base, the method of the present invention can extract key test scenarios more reasonably and accurately, and ensure data privacy while preventing the tedious operation of repeatedly uploading the knowledge base.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle testing technology, and in particular to a method for extracting key test scenarios for intelligent vehicles based on a large language model. Background Art

[0002] Safety testing of smart cars is a fundamental guarantee for their large-scale application. Scenario-based testing methods have advantages over traditional mileage-based methods, such as high testing efficiency and strong repeatability, and have become the focus of existing testing research. Key test scenarios are specific scenarios with a high probability of revealing performance defects in smart car systems, which can cause the tested vehicle to collide or nearly collide with other road entities. Using key test scenarios to test smart cars can improve testing efficiency and reduce costs. However, key test scenarios not only have a "long tail effect", but also the triggering nature of the inducement is unclear, making it difficult to reasonably and accurately extract key test scenarios. The "long tail effect" refers to the fact that the number of key test scenarios is smaller than that of ordinary scenarios, and existing learning or mechanism-based methods find it difficult to efficiently capture the characteristics of key test scenarios under such small sample conditions; the unclear triggering nature of the inducement means that the causes that lead to the occurrence of key test scenarios are extremely complex and difficult to quantify through explicit paradigms or expressions. Large Language Models (LLMs), deep learning models with powerful language understanding and generation capabilities, have been shown to possess exceptionally strong few-shot learning and emergent capabilities. These two advantages offer novel approaches for rationally and accurately extracting key test scenarios. Previous studies have proposed methods for constructing small models based on LLMs, implementing them through parameter fine-tuning and knowledge distillation. Parameter fine-tuning involves fine-tuning and updating pre-trained small models using downstream manually annotated data; knowledge distillation involves training the small model using labels generated by the LLM. However, both approaches have limitations when applied to extracting key test scenarios. The scarcity of actual key test scenarios leads to extremely high costs for manually annotated data, making parameter fine-tuning difficult to effectively extract. Knowledge distillation methods require a large amount of unlabeled data, making acquisition extremely expensive. Furthermore, using online LLMs for extracting key test scenarios presents data privacy issues, and the sheer number of attachments required to build private knowledge bases makes them difficult for online LLMs to support. Summary of the Invention

[0003] In response to the technical problems existing in the application of the above-mentioned large language model to the extraction of key test scenarios, the present invention proposes a key test scenario extraction method for intelligent vehicles based on a large language model, which uses a micro-parameter model to reasonably and accurately extract key test scenarios based on a small sample local private knowledge base.

[0004] The present invention proposes a method for extracting key test scenarios for intelligent vehicles based on a large language model, comprising the following steps:

[0005] Step 1: Real vehicle data collection:

[0006] The data acquisition platform is used to collect real vehicle road image data, and the data format is stored as {x i ,r i ,y i}, where x i is the model input, in the form of image data collected from the real vehicle; r i is a scene description in the form of language and text, obtained through a large language model; i Outputs the criticality of the test scenario for the model in the form of numerical data.

[0007] Step 2: Obtain scene description based on the large language model:

[0008] The image data collected from the real vehicle is input into the large language model, and the large language model is used to obtain the text description corresponding to the image data collected from the real vehicle, and the image and the corresponding description content are stored in the local knowledge base.

[0009] Furthermore, the steps of obtaining scene description based on the large language model include:

[0010] Task introduction: Input task instructions to the large language model to let the large language model understand the purpose of this conversation, which is to determine whether the scene of the image data collected by the real vehicle belongs to the key test scene.

[0011] Analyze scenes: Allow the large language model to freely analyze and describe the input scene image without any restrictions.

[0012] Add guidance: Correct the output of the large language model through subjective description to make it clear why the test scenario is judged as a critical test scenario.

[0013] Explain the reason: Leverage the "emergence ability" of the large language model to fully explain why the input scenario is judged as a key test scenario.

[0014] Preferably, the large language model is the ChatGPT Plus model.

[0015] Step 3: Extract key test scenarios based on the small model:

[0016] First, a manually annotated knowledge base is constructed through subjective evaluation of test scenarios by drivers. Then, a test scenario knowledge base is constructed by selecting typical texts. Finally, a small model for extracting key test scenarios is trained using the manually annotated knowledge base and the test scenario knowledge base, combined with the knowledge base output by the large language model. The model outputs the criticality of the scenarios and the essential causes that lead to the occurrence of key test scenarios.

[0017] Furthermore, extracting key test scenarios based on the small model includes the following steps:

[0018] Step 3.1: Construction of manual annotation knowledge base:

[0019] The video data collected from the actual vehicle is placed on a driving simulator; after observing the video on the driving simulator, the driver who conducts subjective criticality quantification makes a subjective evaluation of the test scene and quantifies the criticality of the test scene.

[0020] Select several different drivers to quantify each test scene, and use weighted method to finally get the manual labeling results of the nth test scene , the specific weighting process is as follows:

[0021] (1)

[0022] Where, j Represents the scene label value 1; m j Select for the driver j The number of drivers corresponding to the label, and the driver's verbal description of the scene are recorded at the same time; M is the number of drivers.

[0023] Step 3.2: Build a test scenario knowledge base:

[0024] Relevant typical texts are selected to build a test scenario knowledge base, including three aspects: key test scenario definitions, key test scenario extraction methods, and scenario criticality quantification methods; the materials that constitute the test scenario knowledge base include test standards, academic papers, and smart car testing regulations.

[0025] Step 3.3: Small model design:

[0026] The pre-trained model input of the small model is the real car collected video data x i , the model output is the criticality y corresponding to the scene i and scene description i ; The knowledge base set obtained by manual annotation is denoted as , the knowledge base set obtained using the large language model is denoted as , and The set of labels that together constitute the model training ;Train small model f The loss function for:

[0027] (2)

[0028] In the formula Represents input x i The corresponding label, ; N represents the number of scenes used for training; r i As f The learning goal is to make the model output the criticality of the scene y i , and also outputs the essential cause r that leads to the occurrence of key test scenarios i , its loss function for:

[0029] (3)

[0030] Where, represent f Output of essential triggers and trigger labels The distance between them.

[0031] (4)

[0032] As a preference, the llava:34b model is selected as the pre-training model for the small model.

[0033] Step 4: Local deployment:

[0034] The model is deployed locally, and before generating an answer, information retrieval is used to search for knowledge related to the question from an external knowledge base. The external knowledge base is composed of scenario descriptions output by a large language model, a manually annotated knowledge base, and a test scenario knowledge base.

[0035] As a preferred option, the Ollama-RAGFlow architecture is used to implement local deployment of the model.

[0036] Beneficial effects of the present invention:

[0037] In response to the technical problems in the process of extracting key test scenarios for smart cars, the present invention proposes a method for extracting key test scenarios for smart cars based on a large language model, which uses a micro-parameter model to reasonably and accurately extract key test scenarios based on a small sample local private knowledge base. First, a large amount of real car scene image data is acquired using a real car data acquisition platform; then, a language description knowledge base corresponding to the scene image data is established using LLM; secondly, labels for scene image data are constructed using manual annotation methods, and a test scene knowledge base is constructed using test standards and other documents, and a local small model training strategy is designed; finally, in order to achieve absolute privacy protection and a personalized knowledge base for the model, the model is deployed locally. Compared with the online LLM that does not establish a local private knowledge base, the method of the present invention can extract key test scenarios more reasonably and accurately, and ensures data privacy while preventing the tedious operation of repeatedly uploading the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall architecture of the present invention.

[0039] Figure 2 Schematic diagram of the real vehicle image data acquisition platform and scene simulation equipment.

[0040] Among them, a is the camera sensor, b is the data storage device, c is the fog simulation device, and d is the rainfall simulation device.

[0041] Figure 3 Different environmental scene information collected from real vehicles.

[0042] Among them, a is a sunny environment, b is a rainy environment, c is a foggy environment, and d is a strong light environment.

[0043] Figure 4 Schematic diagram of the process of obtaining scene description based on a large language model.

[0044] Figure 5 Schematic diagram of the process of building a manually annotated knowledge base.

[0045] Figure 6 Schematic diagram of the method for building a test scenario knowledge base.

[0046] Figure 7 Schematic diagram of the vehicle entering the scene before rainfall and image data collection from the actual vehicle.

[0047] Figure 8 Schematic diagram of image data collection for a rain-following vehicle scenario and a real vehicle.

[0048] Figure 9 Schematic diagram of the scene of the front vehicle entering the foggy weather and the image data collection of the real vehicle.

[0049] Figure 10 Schematic diagram of image data collection for following a vehicle in foggy weather and for a real vehicle.

[0050] Figure 11 Schematic diagram of image data collection for strong light driving scenes and real vehicles.

[0051] Figure 12 This is a comparison chart of the experimental results of the vehicle cutting in before rain.

[0052] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0053] Figure 13 A comparison chart of the experimental results of the rain-following scene.

[0054] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0055] Figure 14 This is a comparison chart of the experimental results of the scene where the front vehicle cuts in on foggy days.

[0056] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0057] Figure 15 A comparison chart of the experimental results of the car-following scenario in foggy weather.

[0058] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0059] Figure 16 This is a comparison chart of the experimental results of the strong light driving scene.

[0060] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention. DETAILED DESCRIPTION

[0061] The present invention provides a method for extracting key test scenarios of intelligent vehicles based on a large language model, the architecture of which is as follows: Figure 1 As shown, the following steps are included:

[0062] Step 1: Real vehicle data collection:

[0063] The real vehicle image data acquisition platform is mainly composed of camera sensors and data storage devices, such as Figure 2 As shown in Figure 1, this acquisition method has the advantages of low cost and high authenticity. The real car scenes collected by the present invention include scene information of rainy environment, foggy environment and strong light environment. In order to obtain scene data of controllable environment, this embodiment chooses to collect data in a closed field with controllable environment, such as Figure 3 shown.

[0064] The data format of the present invention is {x i ,r i ,y i}, where x i The model input is the image data collected from the real vehicle with the same size, which is obtained through the real vehicle data acquisition platform; i It is a scene description in the form of language and text, which is obtained through the subsequent large language model; i The criticality of the test scenario is output to the model in the form of numerical data with a value range of [0,1]. The larger the value, the higher the criticality of the scenario. The true value label is obtained through manual annotation.

[0065] Step 2: Obtain scene description based on the large language model:

[0066] By utilizing the reasoning ability of the large language model, the text description corresponding to the image data collected by the real vehicle is obtained, and the text description is used as the knowledge base of the small model, so that the small model can understand the image data collected by the real vehicle.

[0067] The large language model selected in this embodiment is the ChatGPT Plus model. The image data collected by the real vehicle is input into the large language model, and a text description of the scene is requested. The image and the corresponding description content are stored in the local knowledge base to improve the small model's ability to understand the data collected by the real vehicle.

[0068] The specific process of obtaining scene description based on LLM is as follows Figure 4 As shown, the steps include:

[0069] Task introduction: This process requires the LLM to clarify the purpose of this conversation, which is to determine whether the scene collected by the real vehicle belongs to the key test scene.

[0070] Analyze the scene: Let the LLM analyze the scene image described in the input freely without any restrictions.

[0071] Add guidance: Modify the LLM output through subjective description to make it clear why the test scenario is judged as a critical test scenario.

[0072] Explain the reasons: Leverage the “emergence capability” of LLM to fully explain why the input scenario is judged as a key test scenario.

[0073] Save the above conversation as a document and input it into the local knowledge base of the subsequent small model.

[0074] Step 3: Extract key test scenarios based on the small model:

[0075] Utilizing the knowledge base output by the large language model as labels for training small models results in the extraction of key test scenarios being completely dependent on the performance of the large language model, reducing the accuracy of scenario criticality scoring. Therefore, this paper first constructs a manually annotated knowledge base; then, it constructs a test scenario knowledge base to provide a complete and high-quality knowledge base for the small model; finally, it uses the knowledge base output by the large language model and the manually annotated knowledge base to train the small model for extracting key test scenarios.

[0076] Step 3.1: Construction of manual annotation knowledge base:

[0077] The video data collected from the real vehicle is placed on the driving simulator so that the driver who conducts the subjective evaluation can immersively experience the behavior of the vehicles around the main vehicle; the driver who conducts the subjective criticality quantification needs to make a subjective evaluation of the test scene after observing the video on the driving simulator. The basis for quantification is as follows: Figure 5As shown, the results are obtained based on the evaluation matrix; the evaluation matrix focuses on two aspects of the test scenario: collision risk and whether the driver wants to control the vehicle to improve traffic efficiency. The present invention gives a greater weight to the collision risk dimension in the evaluation matrix; in order to ensure the rationality of subjective evaluation, this embodiment selects 8 different drivers to quantify each test scenario, and finally obtains the manual labeling result of the nth test scenario using a weighted method , the specific weighting process is as follows:

[0078] (1)

[0079] Where, j Represents the scene label value. In this embodiment, four labels are set, and the corresponding values ​​are 0.25, 0.5, 0.75, and 1 respectively; m j Select for the driver j The number of drivers corresponding to the label and the driver's verbal description of the scene are recorded.

[0080] Step 3.2: Build a test scenario knowledge base:

[0081] The data quality of the knowledge base directly affects the application effect of the model. In order to enhance the performance of the small model in the extraction and application of key test scenarios, the present invention selects various relevant typical texts to construct a test scenario knowledge base, which includes three aspects: the definition of key test scenarios, the extraction method of key test scenarios, and the quantification method of the criticality of scenarios. Figure 6 The test scenario knowledge base consists of three aspects: first, test standards, including ISO 26262-1:2011-Roadvehicles-Functional Safety; second, academic papers, including classic Chinese and English academic papers; and finally, smart car testing regulations, including Euro-NCAP and other testing procedures.

[0082] Step 3.3: Small model design:

[0083] The pre-trained model of the small model selects llava:34b, and the model input is the real car collected video data x i , the model output is the criticality y corresponding to the scene i and scene description i The knowledge base set obtained by manual annotation is denoted as ; The knowledge base set obtained using the large language model is denoted as ; and The set of labels that together constitute the model training ;Train small model f The loss function Designed to:

[0084] (2)

[0085] In the formula Represents input x i The corresponding label, ; N represents the number of scenes used for training. i As model input f To train, , in this process r i The acquisition needs to be obtained through LLM or manual annotation. f Therefore, the present invention will r i Likewise as f The learning goal of this model is to not only output the criticality of the scene y i , and can also output the essential causes that lead to the occurrence of key test scenarios i , its loss function Designed to:

[0086] (3)

[0087] Where, represent f Output of essential triggers and trigger labels Therefore, the overall expression of the loss function corresponding to the training of the small model in the present invention is:

[0088] (4)

[0089] Step 4: Local deployment:

[0090] To ensure absolute privacy protection for the model and establish a personalized knowledge base, this paper uses the Ollama-RAGFlow architecture to implement local model deployment. Ollama is an open-source large-scale language model service tool that enables researchers to quickly experiment, manage, and deploy large language models in a local environment. Unlike previous fine-tuning models, RAG (Retrieval-Augmented Generation) searches for question-related knowledge from an external knowledge base through information retrieval before generating an answer, enhancing the information source in the generation process and thus improving the quality and accuracy of the generation. The external knowledge base consists of the scenario description output by the large language model, the manually annotated knowledge base, and the test scenario knowledge base.

[0091] Experimental results analysis:

[0092] The experimental selection analyzes the key test scene extraction method proposed in the present invention from two perspectives: rationality and accuracy. Rationality is used to analyze whether the key test scenes extracted by the present invention method are consistent with the subjective description of manual annotation; accuracy is used to analyze whether the method proposed in the present invention can accurately quantify the criticality of the test scenes. This embodiment selects five scenes for verification experiments, namely, the scene of the car cutting in before the rain (such as Figure 7 As shown in the following scene in rain (as shown in the following scene in rain Figure 8 As shown in the figure), the scene of the car entering the foggy day (as shown in the figure), Figure 9 As shown in the figure), fog following scene (as ... Figure 10 ) and strong light driving scenes (such as Figure 11 The image data collected from the actual vehicle is simultaneously input into the online large language model and the method of the present invention, and the model is required to analyze the criticality of the input scene.

[0093] from Figure 12-16 Experimental results show that compared to online large language models without a private knowledge base, the proposed method not only rationally analyzes the reasons why a scenario is a critical test scenario due to the completeness of the test scenario knowledge base, but also accurately scores the test scenarios based on the manually annotated knowledge base and calculates the average score for each scenario. Comparing the overall scores reveals that the scenarios are ranked in descending order of criticality: Scenario 3 (0.78) > Scenario 4 (0.77) > Scenario 5 (0.76) > Scenario 1 (0.7) > Scenario 2 (0.69). Therefore, Scenario 3 should be given higher priority during smart car testing. Furthermore, compared to online large language models, the proposed method can locally extract key test scenarios, preventing data privacy leaks. Furthermore, through RAGFlow, the proposed method can save all locally uploaded attachments, avoiding the tedious process of repeatedly uploading the knowledge base when starting a new session.

Claims

1. A method for extracting key test scenarios for intelligent vehicles based on a large language model, characterized by: The following steps are involved: Step 1: Real vehicle data collection: The data acquisition platform is used to collect real vehicle road image data, and the data format is stored as {x i ,r i ,y i }, where x i is the model input, in the form of image data collected from the real vehicle; r i is a scene description in the form of language and text, obtained through a large language model; i Output the criticality of the test scenario for the model in the form of numerical data; Step 2: Obtain scene description based on the large language model: Input the image data collected by the real vehicle into the large language model, use the large language model to obtain the text description corresponding to the image data collected by the real vehicle, and store the image and the corresponding description content in the local knowledge base; Step 3: Extract key test scenarios based on the small model: Step 3.1: Constructing a manually annotated knowledge base: Build a manually annotated knowledge base through the subjective evaluation of the test scenarios by the drivers; Step 3.2, test scenario knowledge base construction: Select typical texts to construct the test scenario knowledge base; Step 3.3, small model design: The pre-trained model input of the small model is the image data x collected from the real car i , the model output is the criticality y corresponding to the scene i and scene description i ; The knowledge base set obtained by manual annotation is denoted as , the knowledge base set obtained using the large language model is denoted as , and The set of labels that together constitute the model training ; Loss function for training the small model f for: ； In the formula Represents input x i The corresponding label, ; N represents the number of scenes used for training; r i As the learning goal of f, let the model output the criticality of the scene y i , and also outputs the essential cause r that leads to the occurrence of key test scenarios i , its loss function for: ； Where, Represents the essential inducement and inducement label of f output the distance between them; The overall expression of the loss function corresponding to training the small model is: ； Step 4: Local deployment: The model is deployed locally, and before generating an answer, information retrieval is used to search for question-related knowledge from an external knowledge base. The external knowledge base is composed of scenario descriptions output by a large language model, a manually annotated knowledge base, and a test scenario knowledge base.

2. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 2: Obtaining scene descriptions based on the large language model includes: Task introduction: Input task instructions to the large language model to let it understand the purpose of this conversation, which is to determine whether the scene of the image data collected by the real vehicle belongs to the key test scene; Analyze scenes: Allow the large language model to freely analyze and describe the input scene image without any restrictions; Add guidance: Use subjective descriptions to correct the output of the large language model, so that it can clearly identify the reasons why the test scenario is judged as a critical test scenario; Explain the reason: Leverage the "emergence capability" of the large language model to fully explain why the input scenario is judged as a key test scenario.

3. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1 or 2, characterized in that: The large language model is the ChatGPT Plus model.

4. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 3.1: The method for constructing a manually labeled knowledge base includes: The video data collected from the actual vehicle is placed on the driving simulator; after observing the video on the driving simulator, the driver who conducts subjective criticality quantification makes a subjective evaluation of the test scene and quantifies the criticality of the test scene; Select several different drivers to quantify each test scene, and use weighted method to finally get the manual labeling results of the nth test scene , the specific weighting process is as follows: ； Where j represents the scene label value; m j Select the number of drivers corresponding to label j for the driver, and record the driver's verbal description of the scene; M is the number of drivers.

5. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 3.2: The test scenario knowledge base construction method includes: Relevant typical texts are selected to build a test scenario knowledge base, including three aspects: key test scenario definitions, key test scenario extraction methods, and scenario criticality quantification methods; the materials that constitute the test scenario knowledge base include test standards, academic papers, and smart car testing regulations.

6. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1, characterized in that: The pre-trained model of the small model described in step 3.3 selects the llava:34b model.

7. The method for extracting key test scenarios for intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 4: Local deployment uses the Ollama-RAGFlow architecture to implement local deployment of the model.

Images