Intelligent automobile key test scene extraction method based on large language model

Abstract

The invention provides an intelligent automobile key test scene extraction method based on a large language model, and relates to the technical field of intelligent automobile testing, aiming at solving the technical problems in the intelligent automobile key test scene extraction process. According to the method, the key test scene is reasonably and accurately extracted on the basis of the small sample local private knowledge base by using the microparameter model. The method comprises the steps of real vehicle data acquisition, scene description acquisition based on a large language model, key test scene extraction based on a small model, local deployment and experimental result analysis. Experimental results show that compared with an online LLM without a local private knowledge base, the method has the advantages that the key test scene can be extracted more reasonably and accurately, and data privacy is guaranteed while tedious operation of repeatedly uploading the knowledge base is prevented.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle testing, and particularly relates to a method for extracting key test scenarios of intelligent vehicles based on a large language model. Background Art

[0002] The safety testing of intelligent vehicles is the basic guarantee for their large-scale application. The scenario-based testing method has advantages such as high testing efficiency and strong repeatability compared with the traditional mileage-based method, and has become the focus of existing testing research. Key test scenarios are specific scenarios that have a high probability of revealing performance defects of intelligent vehicle systems, which can lead to collisions or near-collisions between the tested vehicle and other road entities. Using key test scenarios to test intelligent vehicles can improve testing efficiency and reduce costs. However, key test scenarios not only have the "long tail effect", but also the underlying triggering causes are unclear, making it difficult to reasonably and accurately extract key test scenarios. The "long tail effect" means that the number of key test scenarios is relatively small compared to ordinary scenarios, and existing learning-based or mechanism-based methods are difficult to efficiently capture the characteristics of key test scenarios under this small sample condition; the unclear underlying triggering causes mean that the reasons for the occurrence of key test scenarios are extremely complex and difficult to quantify through explicit paradigms or expressions. As a deep learning model with powerful language understanding and generation capabilities, the Large Language Model (LLM) has been pointed out in relevant research to have extremely powerful few-shot learning ability and emergent ability. These two advantages of the LLM provide a new idea for reasonably and accurately extracting key test scenarios. Existing research has proposed methods for constructing small models based on the LLM, and the implementation approaches specifically include parameter fine-tuning and knowledge distillation. Parameter fine-tuning refers to fine-tuning and updating a pre-trained small model using downstream manually annotated data; knowledge distillation refers to training a small model with labels generated by the LLM. However, both of these methods have certain deficiencies when applied to key test scenario extraction. The actual number of key test scenarios that occur is extremely small, resulting in extremely high costs for manually annotated data, and it is difficult for the parameter fine-tuning method to effectively extract; the knowledge distillation method requires a large amount of unlabeled data, resulting in extremely high acquisition costs. In addition, there are data privacy issues when using an online LLM for key test scenario extraction, and the number of attachments for establishing a private knowledge base is huge, and the online LLM is unable to support it. Summary of the Invention

[0003] Aiming at the technical problems existing in the application of the above large language model to key test scenario extraction, the present invention proposes a method for extracting key test scenarios of intelligent vehicles based on a large language model, which reasonably and accurately extracts key test scenarios based on a micro-parameter model on the basis of a small sample local private knowledge base.

[0004] The present invention proposes an intelligent vehicle key test scenario extraction method based on a large language model, including the following steps:

[0005] Step 1, Real vehicle data collection:

[0006] Use a data collection platform to collect real vehicle road image data, and store the data format as {x i , r i , y i}, where x i is the model input, in the form of image data collected from a real vehicle; r i is the scenario description, in the form of language text, obtained through a large language model; y i is the key degree of the model output test scenario, in the form of numerical data.

[0007] Step 2, Obtain scenario description based on a large language model:

[0008] Input the image data collected from the real vehicle into the large language model, and use the large language model to obtain the text description corresponding to the image data collected from the real vehicle, and store the image and the corresponding description content in the local knowledge base.

[0009] Furthermore, the steps of obtaining scenario description based on a large language model include:

[0010] Task introduction: Input a task instruction into the large language model to make the large language model clear the purpose of this conversation, that is, to judge whether the scenario of the image data collected from the real vehicle belongs to a key test scenario.

[0011] Analyze the scenario: Let the large language model freely analyze and describe the input scenario image without any restrictions.

[0012] Add guidance: Modify the output of the large language model through subjective description to make it clear the reason for judging the test scenario as a key test scenario.

[0013] Clarify the reason: Rely on the "emergent ability" of the large language model to fully explain the reason for judging the input scenario as a key test scenario.

[0014] Preferably, the large language model is the ChatGPT Plus model.

[0015] Step 3, Extract key test scenarios based on a small model:

[0016] First, subjectively evaluate the test scenario by the driver to construct an artificial annotation knowledge base; then, select typical texts to construct a test scenario knowledge base; finally, combine the knowledge base output by the large language model, and use the artificial annotation knowledge base and the test scenario knowledge base to train a small model for key test scenario extraction, and output the key degree of the scenario and the essential inducement for the occurrence of the key test scenario.

[0017] Further, the steps for extracting key test scenarios based on a small model include the following:

[0018] Step 3.1: Construction of an artificial annotation knowledge base:

[0019] Arrange the video data collected from a real vehicle on a driving simulator; after observing the video on the driving simulator, the driver who conducts subjective key quantification makes a subjective evaluation of the test scenario to quantify the key degree of the test scenario.

[0020] Select several different drivers to quantify each test scenario, and finally obtain the artificial annotation result of the nth test scenario by using a weighted method , and the specific weighting process is as follows:

[0021] (1)

[0022] In the formula, j represents the scenario label value 1; m j is the number of drivers who select j the label, and at the same time record the driver's verbal description of the scenario; M is the number of drivers.

[0023] Step 3.2: Construction of a test scenario knowledge base:

[0024] Select relevant typical texts to construct a test scenario knowledge base, including three aspects: the definition of key test scenarios, the extraction method of key test scenarios, and the quantification method of scenario key degree; the materials that make up the test scenario knowledge base include test standards, academic papers, and intelligent vehicle test regulations.

[0025] Step 3.3: Design of a small model:

[0026] The input of the pre-training model of the small model is the video data x collected from a real vehicle i , and the output of the model is the key degree y i corresponding to the scenario and the scenario description r i ; the knowledge base set obtained by using artificial annotation is denoted as , and the knowledge base set obtained by using a large language model is denoted as , and together constitute the label set for model training; train the small model f with the loss function as:

[0027] (2)

[0028] In the formula Represents the input x i The corresponding label, ; N represents the number of scenarios used for training; Let r i As f The learning objective of, let the model output the criticality y of the scenario i , and also output the essential inducement r that causes the occurrence of the critical test scenario i , whose loss function Is:

[0029] (3)

[0030] In the formula, Represents f The distance between the output essential inducement and the inducement label Between.

[0031] (4)

[0032] Preferably, the pre-trained model of the small model selects the llava:34b model.

[0033] Step Four: Local Deployment:

[0034] Deploy the model locally, and search for knowledge related to the question from the external knowledge base through information retrieval before generating an answer. The external knowledge base is jointly composed of the scenario descriptions output by the large language model, the manually annotated knowledge base, and the test scenario knowledge base.

[0035] Preferably, the local deployment adopts the Ollama-RAGFlow architecture to realize the local deployment of the model.

[0036] Advantages of the present invention:

[0037] In view of the technical problems in the process of extracting critical test scenarios for intelligent vehicles, the present invention proposes a method for extracting critical test scenarios for intelligent vehicles based on a large language model, which reasonably and accurately extracts critical test scenarios by using a micro-parameter model based on a small-sample local private knowledge base. First, a large amount of real vehicle scenario image data is obtained by using a real vehicle data acquisition platform; then, a language description knowledge base corresponding to the scenario image data is established by using an LLM; secondly, the labels of the scenario image data are constructed by using a manual annotation method, the test scenario knowledge base is constructed by using files such as test standards, and a local small model training strategy is designed; finally, in order to enable the model to achieve absolute privacy protection and a personalized knowledge base, the model is locally deployed. Compared with the online LLM without establishing a local private knowledge base, the method of the present invention can more reasonably and accurately extract critical test scenarios, and while preventing the cumbersome operation of repeatedly uploading the knowledge base, it ensures data privacy. Description of the Drawings

[0038] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0039] Figure 2 This is a schematic diagram of the real vehicle image data acquisition platform and the scene simulation device.

[0040] Among them, a is a camera sensor, b is a data storage device, c is a fog simulation device, and d is a rainfall simulation device.

[0041] Figure 3 These are different environmental scene information collected by the real vehicle.

[0042] Among them, a is a fine weather environment, b is a rainfall environment, c is a fog environment, and d is a strong light environment.

[0043] Figure 4 This is a schematic diagram of the process of obtaining scene descriptions based on large language models.

[0044] Figure 5 This is a schematic diagram of the process of constructing an artificial annotation knowledge base.

[0045] Figure 6 This is a schematic diagram of the method for constructing a test scene knowledge base.

[0046] Figure 7 This is a schematic diagram of the scene of a vehicle cutting in front before rainfall and the real vehicle collected image data.

[0047] Figure 8 This is a schematic diagram of the following vehicle scene during rainfall and the real vehicle collected image data.

[0048] Figure 9 This is a schematic diagram of the scene of a vehicle cutting in front in foggy weather and the real vehicle collected image data.

[0049] Figure 10 This is a schematic diagram of the following vehicle scene in foggy weather and the real vehicle collected image data.

[0050] Figure 11 This is a schematic diagram of the strong light driving scene and the real vehicle collected image data.

[0051] Figure 12 This is a comparison chart of the experimental results of the vehicle cutting in front scene during rainfall.

[0052] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0053] Figure 13 This is a comparison chart of the experimental results of the following vehicle scene during rainfall.

[0054] Among them, a is a schematic diagram of the output result of ChatGPT, and b is a schematic diagram of the output result of the method of the present invention.

[0055] Figure 14 It is a comparison chart of the experimental results for the scenario where the vehicle in front cuts in on a foggy day.

[0056] Among them, a is the schematic diagram of the output result of ChatGPT, and b is the schematic diagram of the output result of the method of the present invention.

[0057] Figure 15 It is a comparison chart of the experimental results for the scenario of following a vehicle on a foggy day.

[0058] Among them, a is the schematic diagram of the output result of ChatGPT, and b is the schematic diagram of the output result of the method of the present invention.

[0059] Figure 16 It is a comparison chart of the experimental results for the scenario of driving in strong light.

[0060] Among them, a is the schematic diagram of the output result of ChatGPT, and b is the schematic diagram of the output result of the method of the present invention. Detailed implementation manner

[0061] An intelligent vehicle key test scenario extraction method based on a large language model provided by the present invention has an architecture as Figure 1 shown, and includes the following steps:

[0062] Step 1, Real vehicle data collection:

[0063] The real vehicle image data collection platform mainly consists of a camera sensor and a data storage device, as Figure 2 shown. This collection method has advantages such as low cost and high authenticity. The real vehicle scenarios collected by the present invention include scenario information in rainfall environment, fog environment, and strong light environment. In order to obtain scenario data in a controllable environment, in this embodiment, data collection is selected in a closed site with a controllable environment, as Figure 3 shown.

[0064] The data format of the present invention is {x i ,r i ,y i}, where x i is the model input, and the specific data form is image data collected from real vehicles with the same size, obtained through the real vehicle data collection platform; r i is the scenario description, in the form of language text, obtained through the subsequent large language model; y i is the key degree of the model output test scenario, in the form of numerical data, and the value range is [0,1]. The larger the value, the higher the key degree of the scenario, and its true value label is obtained through manual annotation.

[0065] Step 2, Obtain the scenario description based on the large language model:

[0066] Utilize the reasoning ability of the large language model to obtain the text description corresponding to the image data collected from the real vehicle, and use this text description as the knowledge base of the small model, enabling the small model to understand the image data collected from the real vehicle.

[0067] In this embodiment, the selected large language model is the ChatGPT Plus model. The image data collected from the real vehicle is input into the large language model, and a request is made for a text description of the scene. The image and the corresponding description content are stored in the local knowledge base to improve the small model's understanding ability of the data collected from the real vehicle.

[0068] The specific process of obtaining the scene description based on the LLM is as Figure 4 shown, and the steps include:

[0069] Task introduction: This process requires the LLM to clarify the purpose of this conversation, that is, to determine whether the scene collected from the real vehicle belongs to a key test scene.

[0070] Analyze the scene: Let the LLM freely analyze and describe the input scene image without any restrictions.

[0071] Add guidance: Modify the output of the LLM through subjective descriptions to make it clear the reasons for judging the test scene as a key test scene.

[0072] Clarify the reasons: Rely on the "emergent ability" of the LLM to fully explain the reasons for judging the input scene as a key test scene.

[0073] Save the above conversation as a document and input it into the local knowledge base of the subsequent small model.

[0074] Step 3. Extract key test scenes based on the small model:

[0075] Fully using the knowledge base output by the large language model as the label for the small model training will lead to the extraction of key test scenes being completely dependent on the performance of the large language model, resulting in a decrease in the accuracy of the scene criticality scoring. Therefore, the present invention first constructs an artificial annotation knowledge base; then, constructs a test scene knowledge base to provide a complete and high-quality knowledge base for the small model; finally, uses the knowledge base output by the large language model and the artificial annotation knowledge base to train a small model for key test scene extraction.

[0076] Step 3.1. Construction of the artificial annotation knowledge base:

[0077] Arrange the video data collected from the real vehicle on the driving simulator so that the driver who conducts the subjective evaluation can immerse himself in the behavior of the vehicles around the host vehicle; after observing the video on the driving simulator, the driver who conducts the subjective key quantification needs to conduct a subjective evaluation of the test scene, and the quantification basis is as Figure 5As shown, the result is obtained according to the evaluation matrix. The evaluation matrix focuses on two aspects of the test scenario: collision risk and whether the driver wishes to control the vehicle to improve traffic efficiency. In the present invention, a greater weight is assigned to the collision risk dimension in the evaluation matrix. To ensure the rationality of the subjective evaluation, in this embodiment, 8 different drivers were selected to quantify each test scenario, and the weighted method was used to finally obtain the manual annotation result of the nth test scenario , and the specific weighting process is as follows:

[0078] (1)

[0079] In the formula, j represents the numerical value of the scenario label. In this embodiment, 4 types of labels are set, and the corresponding numerical values are 0.25, 0.5, 0.75, and 1 respectively; m j is the number of drivers who select the j label, and at the same time, the driver's verbal description of the scenario is recorded.

[0080] Step 3.2, Construction of the test scenario knowledge base:

[0081] The data quality of the knowledge base directly affects the application effect of the model. To enhance the performance of the small model in the extraction and application of key test scenarios, the present invention selects various relevant typical texts to construct a test scenario knowledge base, which includes three aspects: definition of key test scenarios, extraction methods of key test scenarios, and quantification methods of scenario key degrees, as Figure 6 shown. The materials constituting the test scenario knowledge base include three aspects of content. First, the test standards include "ISO 26262-1:2011 - Road vehicles - Functional safety", etc.; second, academic papers, including classic Chinese and English academic papers; finally, intelligent vehicle test regulations, including test procedures such as Euro-NCAP.

[0082] Step 3.3, Design of the small model:

[0083] The pre-trained model of the small model selects llava:34b, and the model input is the video data x collected by the real vehicle i , and the model output is the key degree y i corresponding to the scenario and the scenario description r i . The knowledge base set obtained by manual annotation is denoted as ; the knowledge base set obtained by the large language model is denoted as ; and together constitute the label set for model training; train the small model f with the loss function Designed as:

[0084] (2)

[0085] Wherein represents the input x i corresponding label, ; N represents the number of scenarios used for training. Previous studies often used r i as the model input pair f for training, that is , in this process, the acquisition of r i needs to be obtained through the LLM or manual annotation method, making f the training cost increase. Therefore, the present invention uses r i as f the learning objective as well, which enables the model to not only output the criticality y i of the scenario, but also output the essential cause r i that leads to the occurrence of the critical test scenario. Its loss function is designed as:

[0086] (3)

[0087] Wherein, represents f the distance between the output essential cause and the cause label . Therefore, the overall expression of the loss function for training the small model in the present invention is:

[0088] (4)

[0089] Step Four: Local Deployment:

[0090] In order to enable the model to achieve absolute privacy protection and establish a personalized knowledge base, the present invention uses the Ollama-RAGFlow architecture to implement local deployment of the model; Ollama is an open-source large language model service tool that enables researchers to quickly experiment, manage, and deploy large language models in a local environment; Different from previous fine-tuned models, RAG (Retrieval-Augmented Generation) searches for knowledge related to the question from an external knowledge base through information retrieval before generating an answer, enhancing the information source during the generation process, thereby improving the quality and accuracy of the generation. The external knowledge base is jointly composed of the scenario descriptions output by the large language model, the manually annotated knowledge base, and the test scenario knowledge base.

[0091] Experimental Result Analysis:

[0092] The experiment selects two perspectives to analyze the key test scenario extraction method proposed by the present invention. On the one hand, it is rationality, and on the other hand, it is accuracy. Rationality is used to analyze whether the key test scenarios extracted by the method of the present invention conform to the subjective descriptions manually labeled; accuracy is used to analyze whether the method proposed by the present invention can accurately quantify the key degree of the test scenarios. Five scenarios are selected in this embodiment for verification experiments, namely the pre-rain vehicle cutting-in scenario (as shown in Figure 7 ), the following-vehicle scenario in rain (as shown in Figure 8 ), the pre-fog vehicle cutting-in scenario (as shown in Figure 9 ), the following-vehicle scenario in fog (as shown in Figure 10 ), and the strong-light driving scenario (as shown in Figure 11 ). The image data collected by the real vehicle is input to the online large language model and the method of the present invention at the same time, and the model is required to analyze the key degree of the input scenarios.

[0093] From Figures 12 - 16 the experimental results, it can be seen that compared with the online large language model without building a private knowledge base, the method of the present invention can not only reasonably analyze the reasons for the key test scenarios due to the completeness of the test scenario knowledge base, but also accurately score the test scenarios according to the manually labeled knowledge base, and obtain the overall score corresponding to the scenario by calculating the mean value of the scores in various situations. Through the comparison of the overall scores, it can be obtained that the key degree of the scenarios is ranked from high to low: Scenario 3 (0.78) > Scenario 4 (0.77) > Scenario 5 (0.76) > Scenario 1 (0.7) > Scenario 2 (0.69). Therefore, Scenario 3 should be given a higher priority in the intelligent vehicle test process. In addition, compared with the online large language model, the method designed by the present invention can extract key test scenarios locally, preventing data privacy leakage, and the method of the present invention can save all the attached data uploaded locally through RAGFlow, preventing the cumbersome operation of repeating the upload of the knowledge base when starting a new conversation.

Claims

1. A method for extracting key test scenarios for intelligent vehicles based on a large language model, characterized by: The following steps are involved: Step 1: Real vehicle data collection: The data acquisition platform is used to collect real vehicle road image data, and the data format is stored as {x i ,r i ,y i }, where x i is the model input in the form of image data collected from the real vehicle; r i is a scene description in the form of language and text, obtained through a large language model; i Output the criticality of the test scenario for the model in the form of numerical data; Step 2: Obtain scene description based on the large language model: Input the image data collected by the real vehicle into the large language model, use the large language model to obtain the text description corresponding to the image data collected by the real vehicle, and store the image and the corresponding description content in the local knowledge base; Step 3: Extract key test scenarios based on small models: Step 3.1: Construction of artificial annotation knowledge base: Through the subjective evaluation of the test scene by the driver, the artificial annotation knowledge base is constructed; Step 3.2: Test scenario knowledge base construction: Select typical texts to construct the test scenario knowledge base; Step 3.3, small model design: The pre-trained model input of the small model is the real car collected video data x i , the model output is the criticality y corresponding to the scene i and scene description i ; The knowledge base set obtained by manual annotation is denoted as , the knowledge base set obtained using the large language model is denoted as , and The set of labels that together constitute the model training ; Train a small model f The loss function for: ； In the formula Represents input x i The corresponding label, ; N represents the number of scenes used for training; r i As f The learning goal is to make the model output the criticality of the scene y i , and also outputs the essential causes r that lead to the occurrence of key test scenarios i , its loss function for: ； In the formula, represent f Output of intrinsic incentives and incentive labels The distance between The overall expression of the loss function corresponding to training the small model is: ； Step 4: Local deployment: The model is deployed locally, and before generating an answer, knowledge related to the question is searched from an external knowledge base through information retrieval. The external knowledge base is composed of scenario descriptions output by a large language model, a manually annotated knowledge base, and a test scenario knowledge base.

2. The method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 2: The steps of obtaining scene description based on the large language model include: Task introduction: Input task instructions to the large language model to let the large language model understand the purpose of this conversation, that is, to determine whether the scene of the image data collected by the real car belongs to the key test scene; Analyze scenes: Allow the large language model to freely analyze and describe the input scene images without any restrictions; Add guidance: Correct the output of the large language model through subjective description to make it clear why the test scenario is judged as a key test scenario; Explain the reason: With the help of the "emergence ability" of the large language model, fully explain why the input scenario is judged as a key test scenario.

3. A method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1 or 2, characterized in that: The large language model is the ChatGPT Plus model.

4. The method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 3.1: The method for constructing a manually annotated knowledge base includes: The video data collected from the real vehicle is placed on the driving simulator; after observing the video on the driving simulator, the driver who conducts subjective criticality quantification makes a subjective evaluation of the test scene and quantifies the criticality of the test scene; Select several different drivers to quantify each test scenario, and finally get the first n Manual annotation results of test scenes , the specific weighting process is as follows: ； In the formula, j Represents the scene label value 1; m j Choose for the driver j The number of drivers corresponding to the label, and the driver's verbal description of the scene are recorded; M is the number of drivers.

5. The method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 3.2: The test scenario knowledge base construction method includes: Relevant typical texts are selected to build a test scenario knowledge base, including three aspects: key test scenario definitions, key test scenario extraction methods, and scenario criticality quantification methods; the materials that constitute the test scenario knowledge base include test standards, academic papers, and smart car testing regulations.

6. The method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1, characterized in that: The pre-trained model of the small model described in step 3.3 selects the llava:34b model.

7. The method for extracting key test scenarios of intelligent vehicles based on a large language model according to claim 1, characterized in that: Step 4: Local deployment uses the Ollama-RAGFlow architecture to implement local deployment of the model.

Images