Method for generating high-risk test scene of camera sensing system of autonomous vehicle

Abstract

The invention belongs to the technical field of self-driving automobile testing, and particularly relates to a high-risk test scene generation method for a camera sensing system of a self-driving automobile. Comprising the following steps: step 1, establishing a camera test scene ontology model based on an expected function safety analysis method; 2, establishing a test scene risk assessment model, and generating a high-risk scene ontology by using a model variation test method; and step 3, based on the fine-tuned Open-sora model, realizing conversion between the scene body and the camera perception system test scene, and generating a high-risk test scene video. The method can help an enterprise to establish a perfect automatic driving camera perception system acceleration test system, improves the comprehensiveness and credibility of the enterprise in the test process, can greatly improve the test efficiency based on the high-risk scene acceleration generation test method, reduces the test cost of the enterprise, and improves the test efficiency of the enterprise. And mass production of the automatic driving camera sensing system is accelerated.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle testing technology, specifically a method for generating high-risk test scenarios for autonomous vehicle camera perception systems. Background Technology

[0002] As a core component of autonomous driving systems, the reliable operation of camera perception systems is crucial for ensuring the safety of autonomous vehicles. Currently, scenario-based testing methods are gradually becoming the main means of performance verification for autonomous vehicles. Generating high-risk test scenarios that challenge autonomous driving perception systems is a key factor in ensuring their safety and improving testing efficiency. Existing scholars have conducted some research on camera perception system testing, such as a reliability testing method for autonomous driving perception systems based on fault injection, a testing method and device for intelligent driving perception systems, and a visual injection testing system and method for autonomous driving visual perception systems. However, these methods lack consideration for the video modal input of the test scenario during the camera perception system testing process, and lack consideration for the impact of different scenario elements on the safe operation of the camera perception system. Furthermore, the frequent use of commercial simulation software results in insufficient coverage of scenario elements, making it difficult to quickly and comprehensively identify performance defects in the system under test. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method for generating high-risk test scenarios for autonomous vehicle camera perception systems. This method can help companies establish a comprehensive and accelerated testing system for autonomous vehicle camera perception systems, enhancing the comprehensiveness and reliability of the testing process. Furthermore, the test method based on high-risk scenario acceleration can significantly improve testing efficiency, reduce testing costs, and accelerate the mass production and deployment of autonomous vehicle camera perception systems.

[0004] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0005] This invention provides a method for generating high-risk test scenarios for an autonomous vehicle camera perception system, comprising the following steps:

[0006] Step 1: Based on the expected functional safety analysis method, establish an ontology model of the camera perception system test scenario;

[0007] Based on the ISO 21448 expected functional safety workflow, the failure risks and triggering conditions of camera sensing systems are analyzed, providing theoretical support for the construction of the ontology model of camera sensing system test scenarios. At the same time, the construction of the ontology model of camera sensing system test scenarios based on SOTIF mainly includes four parts: camera sensing system function analysis, system operation hazard event analysis, hazard triggering condition identification, and scene element ontology model construction.

[0008] Step 2: Establish a risk assessment model for the camera perception system test scenario, and use the model mutation testing method to generate a high-risk scenario ontology;

[0009] By systematically constructing typical scenario combinations through combined testing methods, and then applying diverse element variations to the hazard triggering conditions based on these typical scenario combinations, a set of test scenarios is formed.

[0010] Step 3: Based on the fine-tuned Open-sora model, convert the scene text ontology into a video test scene of the camera perception system to generate a high-risk test scene video.

[0011] Furthermore, the specific method for step one is as follows:

[0012] 11) Functional Analysis of Camera Perception System; For autonomous vehicles, the role of the camera perception system is to analyze the surrounding driving conditions, identify the driving area of ​​the autonomous vehicle and potential collision obstacles, i.e., targets; Based on the working principle and functional analysis of the camera, and according to the function of the camera perception system and existing camera testing regulations, the functions of the camera perception system are summarized into four categories: road information recognition (i.e., identifying the driving area of ​​the vehicle), road infrastructure recognition (i.e., identifying the road information status), temporary road status recognition (i.e., identifying unexpected states such as road construction), and traffic target recognition (i.e., identifying surrounding obstacles / target information);

[0013] 12) Hazard event analysis; From the perspective of endangering driving safety, hazard events of the perception system are divided into four categories: target not identified, target classification error, target information identification deviation, and target identification delay.

[0014] 13) Hazard trigger condition identification; establish the mapping relationship between scene elements and hazard triggers using the HAZOP method; focus on the "node" + "parameter" + "guide word" + "consequence" in the HAZOP method, where "node" corresponds to the category of scene element, "parameter" corresponds to the attribute of scene element, "guide word" corresponds to the abnormal characteristics of scene element, and "consequence" corresponds to the system hazard triggered; the hazard trigger conditions of the camera perception system based on HAZOP include five types of nodes: light source, target object, medium, lens, and photosensitive element;

[0015] 14) Scene element ontology model construction; the scene is described in the form of an ontology; the key test scene ontology data is divided into two parts: scene basic information data and hazard triggering data. The scene basic information data is the combination of basic elements that constitute the scene, and only the basic elements that constitute the scene are considered; the hazard triggering data refers to the types and values ​​of elements that may trigger scene hazards, which are converted from guide words; the Neo4j software is used to realize scene information storage based on knowledge graph, and the semantic Web rule language is introduced into the ontology to realize knowledge reasoning based on ontology.

[0016] For the basic information data of the scene, it is constructed based on the six-layer model of the scene; the six-layer model of the scene includes (1) road layer: road network, geometric structure, road surface quality; (2) infrastructure layer: traffic signs, traffic lights; (3) temporary operations of the first two layers: such as traffic control, road construction; (4) object layer: traffic participant behavior and status; (5) environment layer: weather, lighting; (6) digital information layer: digital information; the temporary nature of the third layer of roads and facilities is classified into the hazard trigger data, while the sixth layer of digital information is not considered. The final basic information data of the scene includes four categories: roads, infrastructure, objects, weather and lighting.

[0017] Furthermore, the specific method for step two is as follows:

[0018] 21) Suppose the test scenario ontology description consists of n key elements, each element x i Values ​​are taken from the discrete set D i Then any scene ontology is represented as:

[0019]

[0020] For each element set D i Divide the system into a subset D representing the system design conditions or typical combinations. i type This allows for the construction of a typical scene combination space:

[0021]

[0022] 22) Generate the typical scenario combination space S in formula (2) using a greedy combinatorial testing method. base ;

[0023] 23) After completing the combination of basic scenario information data, a model mutation testing mechanism is introduced to incorporate hazard triggering data into the construction of the scenario ontology, increasing the testing difficulty and criticality of the scenario; let the typical test scenario ontology model be represented as:

[0024] M:S→Y (3)

[0025] In the formula, Y represents the difficulty and criticality of the scenario test;

[0026] 24) The criticality of the scenario test is determined by superimposing the criticality of the elements. First, the criticality value of each scenario element is determined by the analytic hierarchy process. Each scenario element and its value are compared pairwise to obtain the scenario element importance comparison matrix A, as shown in formula (4):

[0027]

[0028] In the formula, a nn To compare the relative importance of different elements;

[0029] Then the eigenvectors of the matrix in formula (4) are calculated and the consistency ratio is checked. If the check is passed, the calculated maximum eigenvalue is used as the key weight of the element. Otherwise, the importance comparison matrix (4) is reconstructed and calculated until the consistency ratio meets the requirements.

[0030] 25) After obtaining the importance of different scene elements, the combined scene key quantitative indicators are as follows:

[0031]

[0032] In the formula, I i The importance of the value of the i-th element;

[0033] Define a set of mutation operators consisting of hazard-triggered data. Each operator acts on the input or model structure, forming a mutated model or mutated scenario input, resulting in the following mutated output:

[0034] y (k) =T k (s)=M (k) (s (k) (6)

[0035] In the formula, M (k) =δ k (M) represents a variation in scene parameter type, meaning the addition of scene elements not included in the original typical scene; s (k) =δ k (s) is to change the values ​​of certain elements to accident-triggered elements based on the original typical scenario;

[0036] 26) In the process of scene generation, the basic use case S is first selected from typical scenarios. base Apply mutation operator set {δ k}, forming a test set:

[0037] S gen ={δ k (s i )|s i ∈S base ,δ k ∈△}(7)

[0038] In the formula, Δ represents the effective mutation operator set;

[0039] Introduce a threshold function for scene criticality metrics:

[0040] D raise(y i ,y i (k) )=y i -y i (k) (8)

[0041] D total (y i (k) )=y i (k) -y threshold (9)

[0042] In the formula, D raise This describes the key improvements of the mutated scene compared to the original scene; D total This represents the improvement in the scene's criticality threshold after the mutation; if D raise Or D total If any metric meets the set conditions, the modified test case is considered to have increased the difficulty, and the generated test case is the key scenario ontology.

[0043] 27) Set a mutation dimension coverage index to further quantify the spatial diversity and mutation intensity of the test set;

[0044]

[0045] In the formula, MT ac The mutation dimension activated for the current use case; MT total The number of all mutable dimensions.

[0046] Furthermore, the specific method for step three is as follows:

[0047] 31) Use a large language model for text-to-video conversion to automatically convert the test scene content described in natural language into a video clip with semantic coherence, visual consistency and temporal dynamics.

[0048] The text-to-video conversion process is shown in formula (11):

[0049] T→V, V={F1,F2,…,F n} (11)

[0050] In the formula, T represents the natural language description; V represents a video segment automatically generated by the large model that is consistent with the semantics; and Fn represents different video frames.

[0051] The current application process of the large model is to fine-tune the general large model; the fine-tuning dataset is the core of the training effect of the Open-Sora model. The Road Hazard Stimuli database, MP-RAD database, and BDD100K database are used as the basic data for fine-tuning. The three types of databases correspond to the corresponding hazard triggering conditions.

[0052] 32) Randomly select 200 video data from three databases: Road Hazard Stimuli, MP-RAD, and BDD100K. Use Monte Carlo simulation to generate photosensor failure and noise detection scenarios for 100 videos from each database.

[0053] 33) Based on the obtained video data, the first step is to annotate the prompt words and write rough text description prompt words for each video segment. The rough description needs to accurately summarize the key scene elements and temporal processes contained in the video. The scene elements need to be consistent with the basic scene information data and hazard trigger data in the defined scene ontology. Then, the Video-LLaMA 2 model for multimodal data understanding is introduced to refine and enhance the rough-annotated prompt words in one stage. The videos in the database and the annotated rough prompt words are input into the Video-LLaMA 2 model. The visual semantic understanding ability is used to generate more detailed and accurate text descriptions and correct the descriptions that do not match the picture, so as to obtain the refined and enhanced prompt words.

[0054] 34) The enhanced cue words obtained and the original video data constitute the training sample pairs (T) train V train The process corresponds to the text-to-video conversion process in formula (11); based on the obtained (T) train V train The Open-Sora model parameters θ are fine-tuned and trained by maximizing the log-likelihood of the real video under text conditions, as shown in Equation (12):

[0055]

[0056] In the formula, p θ (V train |T train ) is used to fine-tune the model with text T train Generate video V train The probability distribution is optimized by minimizing the difference between the generated video and the real video, so that the model can gradually learn to generate realistic transition videos under the prompts of accident text descriptions.

[0057] 35) After fine-tuning the Open-Sora model, the corresponding key test scene video data of the camera perception system can be generated according to the text description. First, the required key test scene ontology description is generated using formulas (1) to (7), which corresponds to the stage of manually writing prompt words in the training process of the Open-Sora large model. Then, the prompt word refinement process is designed, and the generated scene ontology description is expanded into a more detailed and refined enhanced prompt word Ten using the ChatGPT-4o large language model. In the enhancement process, it is necessary to ensure that the expanded text description covers all the key information in the generated scene ontology, and at the same time add necessary contextual details to make the description more coherent. Then, Ten is input into the fine-tuned Open-Sora model to generate test scene videos that match the refined prompt word description.

[0058] The beneficial effects of this invention are as follows:

[0059] The results of this invention can help enterprises establish a comprehensive accelerated testing system for autonomous driving camera perception systems, improve the comprehensiveness and credibility of the enterprise's testing process, and at the same time, the test method based on high-risk scenarios can greatly improve testing efficiency, reduce enterprise testing costs, and accelerate the mass production and deployment of autonomous driving camera perception systems. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of the test scene for a SOTIF-based camera perception system.

[0062] Figure 2 A schematic diagram of the defined ontology entities and their relationships;

[0063] Figure 3 This is a schematic diagram of photosensitive failure generated for 100 videos each using Monte Carlo simulation.

[0064] Figure 4 This is a schematic diagram of noise detection generated for 100 videos each using Monte Carlo simulation.

[0065] Figure 5 This is a partial example diagram of the ontology content;

[0066] Figure 6 A diagram illustrating the before-and-after comparison of enhancing prompt words;

[0067] Figure 7 This is a schematic diagram of partial frames in a test scene of a camera perception system video generated using the finely tuned Open-Sora. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0069] Example 1

[0070] This embodiment provides a method for generating high-risk test scenarios for an autonomous vehicle camera perception system, including the following steps:

[0071] Step 1: Based on the expected functional safety analysis method, establish an ontology model of the camera test scenario;

[0072] Based on the ISO 21448 expected functional safety workflow, this paper analyzes the failure risks and triggering conditions of camera sensing systems, providing theoretical support for the construction of an ontology model of camera sensing system test scenarios. Meanwhile, the construction of the ontology model of camera sensing system test scenarios based on SOTIF mainly includes four parts: camera sensing system functional analysis, system operation hazard event analysis, hazard triggering condition identification, and scene element ontology model construction, as detailed below:

[0073] 11) Considering the complexity and unpredictability of the camera sensing system's operation, the failure risks and triggering conditions of the camera sensing system are analyzed based on the ISO 21448 expected functional safety workflow, providing theoretical support for the construction of the camera sensing system test scenario ontology model. Simultaneously, to ensure the standardization and planning of test scenario descriptions, and to facilitate the uniformity of subsequent large-scale model scene generation processes, a unified scene ontology is used for the camera sensing system ontology description. The construction of the camera sensing system test scenario ontology based on SOTIF mainly includes four parts: camera sensing system functional analysis, system operation hazard event analysis, hazard triggering condition identification, and scene element ontology model construction, such as... Figure 1 As shown.

[0074] 111) Functional Analysis of Camera Perception System. For autonomous vehicles, the main function of the camera perception system is to analyze the surrounding driving conditions, identify the drivable area of ​​the autonomous vehicle, and identify potential collision obstacles (objects). Based on the working principle and functional analysis of cameras, and according to the functions of the camera perception system and existing camera testing regulations, the functions of the camera perception system can be summarized into four main categories: road information recognition (identifying the vehicle's drivable area), road infrastructure recognition (identifying road information status), temporary road condition recognition (identifying unexpected conditions such as road construction), and traffic object recognition (identifying surrounding obstacles / object information). Specific functions are shown in Table 1.

[0075] Table 1 Camera Sensing System Functions

[0076]

[0077] 112) Hazard event analysis. From the perspective of endangering driving safety, hazard events of the perception system can be divided into four categories: target not identified, target classification error, target information identification deviation, and target identification delay. The corresponding explanations are shown in Table 2.

[0078] Table 2 Hazardous Events of Camera Sensing Systems

[0079]

[0080] 113) Hazard Trigger Identification. The HAZOP method is used to establish a mapping relationship between scene elements and hazard triggers. The HAZOP method mainly adopts a structure of nodes, parameters, guiding words, consequences, protective measures, and recommended measures. For the construction process of the camera perception system test scene ontology, its main purpose is to analyze the system hazard trigger conditions and thus determine key scene elements. Therefore, the focus is on the "node" + "parameter" + "guiding word" + "consequence" in the HAZOP method, where "node" corresponds to the category of scene element, "parameter" corresponds to the attribute of scene element, "guiding word" corresponds to the abnormal characteristics of scene element, and "consequence" corresponds to the triggered system hazard. Based on the working principle of the camera (… Figure 1 Considering both the camera sensing system functions in Table 1 and the camera sensing system hazard events in Table 2, the hazard triggering conditions for the camera sensing system based on HAZOP are summarized in Table 3. These include five types of nodes: light source, target object, medium, lens, and photosensitive element.

[0081] Table 3. Hazard Triggering Conditions for Camera Sensing Systems

[0082]

[0083]

[0084]

[0085] 114) Scene Element Ontology Model Construction. To ensure the standardization of scene element descriptions and facilitate subsequent scene semantic generation and video generation, an ontology is used to describe the scene. The key test scene ontology data is divided into two parts: basic scene information data and hazard triggering data. Basic scene information data refers to the combination of basic elements constituting the scene; this part only considers the basic elements constituting the scene and does not consider abnormal element values. Hazard triggering data refers to the types and values ​​of elements that may trigger scene hazards, converted from the guiding words in Table 3. Neo4j software is used to implement knowledge graph-based scene information storage, and semantic Web rule language is introduced into the ontology to achieve ontology-based knowledge reasoning. The defined ontology entities and relationships are as follows: Figure 2 As shown.

[0086] The basic information data of the scene is constructed based on the six-layer model of the scene. The six-layer model of the scene mainly includes (1) road layer: road network, geometric structure, road surface quality; (2) infrastructure layer: traffic signs, traffic lights, etc.; (3) temporary operations of the first two layers: such as traffic control, road construction, etc.; (4) object layer: behavior and status of traffic participants; (5) environment layer: weather, lighting and other environmental conditions; (6) digital information layer: digital information, such as V2X information, etc. The temporary operations of the third layer of roads and facilities will cause great interference to the recognition of the perception system when they occur, so they are classified into the hazard trigger data. At the same time, the sixth layer of digital information layer is not considered. The final basic information data of the scene includes four categories: roads, infrastructure, objects, weather and lighting.

[0087] for Figure 2 The attribute values ​​(scene element values) of different entities are set with reference to "GB 5768 Road Traffic Signs and Markings", "GB / T 45312-2025 Design and Operation Conditions for Automated Driving Systems of Intelligent Connected Vehicles", and "GB / T 41798-2022 Site Test Methods and Requirements for Automated Driving Functions of Intelligent Connected Vehicles".

[0088] Step 2: Establish a risk assessment model for the camera perception system test scenario, and use the model mutation testing method to generate a high-risk scenario ontology;

[0089] By systematically constructing typical scenario combinations using a combined testing method, and then applying diverse element variations to the hazard triggering conditions based on these typical scenario combinations, a set of test scenarios is formed, as detailed below:

[0090] To improve the robustness and boundary capability verification of camera perception systems in various complex scenarios, this invention proposes a key test scenario ontology generation method that combines scene basic information data combination generation with a hazard triggering data mutation testing mechanism. This method systematically constructs typical scene combinations through a combination testing approach, and then applies diverse element mutations based on hazard triggering conditions to form a well-structured test scenario set with hazard triggering potential.

[0091] 21) Suppose the test scenario ontology description consists of n key elements, each element x i Values ​​are taken from the discrete set D i Then any scene ontology can be represented as:

[0092]

[0093] For each element set D i Further subdividing the values ​​into subsets D that represent the system design conditions or typical combinations i type This allows for the construction of a typical scene combination space:

[0094]

[0095] 22) Generate the typical scenario combination space S in formula (2) using a greedy combinatorial testing method. base The pseudocode for the greedy combinatorial testing method is shown in Code 1.

[0096]

[0097]

[0098] 23) After completing the combination of basic scenario information data, a model mutation testing mechanism is introduced to incorporate hazard triggering data into the construction of the scenario ontology, increasing the testing difficulty and criticality of the scenario. Let the typical test scenario ontology model be represented as:

[0099] M:S→Y (3)

[0100] In the formula, Y represents the difficulty and criticality of the scenario test.

[0101] 24) The criticality of the scenario test is determined by superimposing the criticality of the elements. First, the criticality value of each scenario element is determined by the hierarchical analysis method. Each scenario element and its value are compared in pairs to obtain the scenario element importance comparison matrix A, as shown in formula (4).

[0102]

[0103] In the formula, a nnThis refers to the relative importance of different elements.

[0104] Then, the eigenvectors of the matrix in formula (4) are calculated and the consistency ratio is checked. If the check is passed, the calculated maximum eigenvalue is used as the key weight of the element. Otherwise, the importance comparison matrix (4) is reconstructed and calculated until the consistency ratio meets the requirements.

[0105] 25) After obtaining the importance of different scene elements, the combined quantitative indicators of scene keyness can be obtained as follows:

[0106]

[0107] In the formula, I i The importance of the value of the i-th element.

[0108] Define a set of mutation operators consisting of hazard-triggered data. Each operator acts on the input or model structure, forming a mutated model or mutated scenario input, resulting in the following mutated output:

[0109] y (k) =T k (s)=M (k) (s (k) (6)

[0110] Among them, M (k) =δ k (M) indicates a variation in scene parameter type, meaning the addition of scene elements not included in the original typical scene, such as... Figure 2 Road construction information in s (k) =δ k (s) indicates that the values ​​of certain elements are changed to accident-triggered element values ​​based on the original typical scenario, such as changing the pedestrian in the typical scenario to a pedestrian with an umbrella.

[0111] 26) In the process of scene generation, the basic use case S is first selected from typical scenarios. base Apply the set of mutation operators {δ} to it k}, forming a test set:

[0112] S gen ={δ k (s i )|s i ∈S base ,δ k ∈△}(7)

[0113] Where Δ represents the effective mutation operator set.

[0114] To measure the effectiveness of the applied variations in the test, a threshold function for scene criticality is introduced:

[0115] D raise (y i ,y i (k) )=y i -y i (k) (8)

[0116] D total (y i (k) )=y i (k) -y threshold (9)

[0117] In the formula, D raise D represents the key improvements of the mutated scene compared to the original scene. total To show the improvement in scene criticality threshold after mutation, if D raise Or D total If any one of the indicators meets the set conditions, it can be considered that the mutated test cases have increased in difficulty, and the generated test cases are the key scenario ontology.

[0118] 27) In addition, a mutation dimension coverage index is set to further quantify the spatial diversity and mutation intensity of the test set;

[0119]

[0120] In the formula, MT ac MT is the mutation dimension activated for the current use case. total The number of all mutable dimensions.

[0121] Step 3: Based on the fine-tuned Open-sora model, convert the scene text ontology into a video test scene from the camera perception system to generate a high-risk test scene video, as detailed below:

[0122] 31) Use a large language model for text-to-video conversion to automatically convert the test scene content described in natural language into a video clip with semantic coherence, visual consistency and temporal dynamics.

[0123] The text-to-video conversion process is shown in formula (11):

[0124] T→V, V={F1,F2,…,F n} (11)

[0125] In the formula, T represents the natural language description, V represents a video segment automatically generated by the large model that is consistent with the semantics, and Fn represents different video frames.

[0126] The currently trained general-purpose large model already possesses general video conversion capabilities, but it may have certain shortcomings when performing specific tasks. Considering the computational resources consumed in the complete training process of the large model, the current application process mainly adopts the method of fine-tuning the general-purpose large model. The fine-tuning dataset is the core factor affecting the training effect of the Open-Sora model. The Road Hazard Stimuli database, MP-RAD database, and BDD100K database are used as the basic data for fine-tuning. The introduction of the three types of databases and the corresponding hazard triggering conditions are shown in Table 2.

[0127] Table 4. Open-Sora fine-tuning database used in this invention.

[0128]

[0129] 32) Regarding the photosensitive element hazard triggering conditions in Table 3, this paper randomly selected 200 video data from the above three types of databases, and generated photosensitive failure and noise detection scenarios for 100 videos in each category using Monte Carlo simulation. Photosensitive failure ( Figure 3 (Mid-black point) and detection noise ( Figure 4 Example of the generated effect of white dot (in the middle) is as follows: Figure 3 and Figure 4 As shown.

[0130] 33) Based on the video data obtained above, manual tagging of prompts is first performed. Rough text description prompts are written manually for each video segment. These rough descriptions need to accurately summarize the key scene elements and temporal sequence contained in the video. The scene elements need to be consistent with… Figure 2 The scene basic information data and hazard trigger data defined in the scene ontology are consistent. However, relying solely on manual annotation may lead to problems such as omissions of details or inconsistencies in expression style. Therefore, Video-LLaMA 2, a large model for multimodal data understanding, is introduced to refine and enhance the coarsely annotated prompts. The coarsely annotated prompts from the videos in the database in Table 4 and the manually annotated prompts are input into the Video-LLaMA 2 model. Its visual semantic understanding capabilities are used to generate more detailed and accurate text descriptions, supplementing environmental details, traffic participant actions, etc., that were missed by manual annotations, and correcting descriptions that do not match the images, thus obtaining finely adjusted and enhanced prompts.

[0131] 34) The enhanced cue words obtained and the original video data constitute the training sample pairs (T) train V train This process corresponds to the text-to-video conversion process in formula (11). Based on the obtained (T) train V trainThe Open-Sora model parameters θ are fine-tuned and trained by maximizing the log-likelihood of the real video under text conditions, as shown in Equation (12):

[0132]

[0133] In the formula, p θ (V train |T train ) indicates that the model is fine-tuned using text T train Generate video V train The probability distribution is used to optimize the model by minimizing the difference between the generated video and the real video, so that the model can gradually learn to generate realistic transition videos based on accident text description prompts.

[0134] 35) After fine-tuning the Open-Sora model, the corresponding key test scene video data of the camera perception system can be generated based on the text description. First, the required key test scene ontology description is generated using formulas (1) to (7). This process corresponds to the stage of manually writing prompts during the training process of the Open-Sora large model. Since this stage only contains key scene elements in the scene ontology and some scene elements have empty values, it is difficult to guide the fine-tuned Open-Sora model to generate high-fidelity and highly controllable test scene videos. Therefore, this paper designs a prompt refinement process, which uses the ChatGPT-4o large language model to expand the generated scene ontology description into a more detailed and refined enhanced prompt word Ten. During the enhancement process, it is necessary to ensure that the expanded text description covers all the key information in the generated scene ontology, and at the same time add necessary contextual details to make the description more coherent. Then, Ten is input into the fine-tuned Open-Sora model to generate test scene videos that match the refined prompt word description.

[0135] Example 2

[0136] This embodiment initially generated 500 key test scene ontology entries that pose a challenge to the camera perception system by setting scene criticality metric thresholds. Examples of some ontology content are shown below. Figure 5 As shown (searching for heavy snow scenes).

[0137] After generating the key test scene ontology, the scene ontology information is simply combined to form coarse cue words. Then, ChatGPT-4o is used to generate refined, logically consistent cue words from the generated coarse scene ontology, supplementing the coherent descriptions in the scene-generated cue words. This guides the fine-tuned Open-Sora model in generating corresponding videos. The comparison of the cue word enhancement before and after is shown below. Figure 6 As shown.

[0138] by Figure 6 Taking the prompt words in the video as an example, the effect of generating a partial frame of the video using the finely tuned Open-Sora is as follows: Figure 7 As shown.

[0139] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating high-risk test scenarios for an autonomous vehicle camera perception system, characterized in that, Includes the following steps: Step 1: Based on the expected functional safety analysis method, establish an ontology model of the camera test scenario; Based on the ISO 21448 expected functional safety workflow, the failure risks and triggering conditions of camera sensing systems are analyzed, providing theoretical support for the construction of the ontology model of camera sensing system test scenarios. At the same time, the construction of the ontology model of camera sensing system test scenarios based on SOTIF mainly includes four parts: camera sensing system function analysis, system operation hazard event analysis, hazard triggering condition identification, and scene element ontology model construction. Step 2: Establish a risk assessment model for the camera perception system test scenario, and use the model mutation testing method to generate a high-risk scenario ontology; By systematically constructing typical scenario combinations through combined testing methods, and then applying diverse element variations to the hazard triggering conditions based on these typical scenario combinations, a set of test scenarios is formed. Step 3: Based on the fine-tuned Open-sora model, convert the scene text ontology into a video test scene of the camera perception system to generate a high-risk test scene video.

2. The method for generating high-risk test scenarios for an autonomous vehicle camera perception system according to claim 1, characterized in that, The specific method for step one is as follows: 11) Functional Analysis of Camera Perception System; For autonomous vehicles, the role of the camera perception system is to analyze the surrounding driving conditions, identify the driving area of ​​the autonomous vehicle and potential collision obstacles, i.e., targets; Based on the working principle and functional analysis of the camera, and according to the function of the camera perception system and existing camera testing regulations, the functions of the camera perception system are summarized into four categories: road information recognition (i.e., identifying the driving area of ​​the vehicle), road infrastructure recognition (i.e., identifying the road information status), temporary road status recognition (i.e., identifying unexpected states such as road construction), and traffic target recognition (i.e., identifying surrounding obstacles / target information); 12) Hazard event analysis; From the perspective of endangering driving safety, hazard events of the perception system are divided into four categories: target not identified, target classification error, target information identification deviation, and target identification delay. 13) Hazard trigger condition identification; establish the mapping relationship between scene elements and hazard triggers using the HAZOP method; focus on the "node" + "parameter" + "guide word" + "consequence" in the HAZOP method, where "node" corresponds to the category of scene element, "parameter" corresponds to the attribute of scene element, "guide word" corresponds to the abnormal characteristics of scene element, and "consequence" corresponds to the system hazard triggered; the hazard trigger conditions of the camera perception system based on HAZOP include five types of nodes: light source, target object, medium, lens, and photosensitive element; 14) Scene element ontology model construction; the scene is described in the form of ontology; the key test scene ontology data is divided into two parts: scene basic information data and hazard triggering data. The scene basic information data is the combination of basic elements that constitute the scene; the hazard triggering data refers to the types and values ​​of elements that may trigger scene hazards, which are transformed from guide words; the Neo4j software is used to realize scene information storage based on knowledge graph, and the semantic Web rule language is introduced into the ontology to realize knowledge reasoning based on ontology. For the basic information data of the scene, it is constructed based on the six-layer model of the scene; the six-layer model of the scene includes (1) road layer: road network, geometric structure, road surface quality; (2) infrastructure layer: traffic signs, traffic lights; (3) temporary operations of the first two layers: such as traffic control, road construction; (4) object layer: traffic participant behavior and status; (5) environment layer: weather, lighting; (6) digital information layer: digital information; the temporary nature of the third layer of roads and facilities is classified into the hazard trigger data, while the sixth layer of digital information is not considered. The final basic information data of the scene includes four categories: roads, infrastructure, objects, weather and lighting.

3. The method for generating high-risk test scenarios for an autonomous vehicle camera perception system according to claim 1, characterized in that, The specific method for step two is as follows: 21) Suppose the test scenario ontology description consists of n key elements, each element x i Values ​​are taken from the discrete set D i Then any scene ontology is represented as: For each element set D i Divide the system into a subset D representing the system design conditions or typical combinations. i type This allows for the construction of a typical scene combination space: 22) Generate the typical scenario combination space S in formula (2) using a greedy combinatorial testing method. base ; 23) After completing the combination of basic scenario information data, a model mutation testing mechanism is introduced to incorporate hazard triggering data into the construction of the scenario ontology, increasing the testing difficulty and criticality of the scenario; let the typical test scenario ontology model be represented as: M:S→Y (3) In the formula, Y represents the difficulty and criticality of the scenario test; 24) The criticality of the scenario test is determined by superimposing the criticality of the elements. First, the criticality value of each scenario element is determined by the analytic hierarchy process. Each scenario element and its value are compared pairwise to obtain the scenario element importance comparison matrix A, as shown in formula (4): In the formula, a nn To compare the relative importance of different elements; Then the eigenvectors of the matrix in formula (4) are calculated and the consistency ratio is checked. If the check is passed, the calculated maximum eigenvalue is used as the key weight of the element. Otherwise, the importance comparison matrix (4) is reconstructed and calculated until the consistency ratio meets the requirements. 25) After obtaining the importance of different scene elements, the combined scene key quantitative indicators are as follows: In the formula, I i The importance of the value of the i-th element; Define a set of mutation operators consisting of hazard-triggered data. Each operator acts on the input or model structure, forming a mutated model or mutated scenario input, resulting in the following mutated output: y (k) =T k (s)=M (k) (s (k) ) (6) In the formula, M (k) =δ k (M) represents a variation in scene parameter type, meaning the addition of scene elements not included in the original typical scene; s (k) =δ k (s) is to change the values ​​of certain elements to accident-triggered elements based on the original typical scenario; 26) In the process of scene generation, the basic use case S is first selected from typical scenarios. base Apply mutation operator set {δ k }, forming a test set: S gen ={δ k (s i )|s i ∈S base ,d k ∈△}(7) In the formula, Δ represents the effective mutation operator set; Introduce a threshold function for scene criticality metrics: D raise (and i ,and i (k) )=and i -and i (k) (8) D total (and i (k) )=and i (k) -and threshold (9) In the formula, D raise This describes the key improvements of the mutated scene compared to the original scene; D total This represents the improvement in the scene's criticality threshold after the mutation; if D raise Or D total If any metric meets the set conditions, the modified test case is considered to have increased the difficulty, and the generated test case is the key scenario ontology. 27) Set a mutation dimension coverage index to further quantify the spatial diversity and mutation intensity of the test set; In the formula, MT ac The mutation dimension activated for the current use case; MT total The number of all mutable dimensions.

4. The method for generating high-risk test scenarios for an autonomous vehicle camera perception system according to claim 3, characterized in that, The specific method for step three is as follows: 31) Use a large language model for text-to-video conversion to automatically convert the test scene content described in natural language into a video clip with semantic coherence, visual consistency and temporal dynamics. The text-to-video conversion process is shown in formula (11): T→V,V={F1,F2,…,F n } (11) In the formula, T represents the natural language description; V represents a video segment automatically generated by the large model that is consistent with the semantics; and Fn represents different video frames. The current application process of the large model is to fine-tune the general large model; the fine-tuning dataset is the core of the training effect of the Open-Sora model. The Road Hazard Stimuli database, MP-RAD database, and BDD100K database are used as the basic data for fine-tuning. The three types of databases correspond to the corresponding hazard triggering conditions. 32) Randomly select 200 video data from three databases: Road Hazard Stimuli, MP-RAD, and BDD100K. Use Monte Carlo simulation to generate photosensor failure and noise detection scenarios for 100 videos from each database. 33) Based on the obtained video data, the first step is to annotate the prompt words and write rough text description prompt words for each video segment. The rough description needs to accurately summarize the key scene elements and temporal processes contained in the video. The scene elements need to be consistent with the basic scene information data and hazard trigger data in the defined scene ontology. Then, the Video-LLaMA 2 model for multimodal data understanding is introduced to refine and enhance the rough-annotated prompt words in one stage. The videos and the annotated rough prompt words in the database are input into the Video-LLaMA 2 model. The visual semantic understanding ability is used to generate more detailed and accurate text descriptions and correct the descriptions that do not match the picture, so as to obtain the refined and enhanced prompt words. 34) The enhanced cue words obtained and the original video data constitute the training sample pairs (T) train V train The process corresponds to the text-to-video conversion process in formula (11); based on the obtained (T) train V train The Open-Sora model parameters θ are fine-tuned and trained by maximizing the log-likelihood of the real video under text conditions, as shown in Equation (12): In the formula, p θ (V train |T train ) is used to fine-tune the model with text T train Generate video V train The probability distribution is optimized by minimizing the difference between the generated video and the real video, so that the model can gradually learn to generate realistic transition videos under the prompts of accident text descriptions. 35) After fine-tuning the Open-Sora model, the corresponding key test scene video data of the camera perception system can be generated according to the text description. First, the required key test scene ontology description is generated using formulas (1) to (7), which corresponds to the stage of manually writing prompt words in the training process of the Open-Sora large model. Then, the prompt word refinement process is designed, and the generated scene ontology description is expanded into a more detailed and refined enhanced prompt word Ten using the ChatGPT-4o large language model. In the enhancement process, it is necessary to ensure that the expanded text description covers all the key information in the generated scene ontology, and at the same time add necessary contextual details to make the description more coherent. Then, Ten is input into the fine-tuned Open-Sora model to generate test scene videos that match the refined prompt word description.

Images