Method for constructing multi-modal accident scene library for end-to-end autonomous driving test

Abstract

This invention belongs to the field of autonomous driving testing technology, specifically a method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing. The method includes the following steps: Step 1, extracting text information based on the MCAT-BiLSTM-CRF algorithm; Step 2, designing an ontology architecture and using a knowledge graph to store accident scenario information, resulting in an accident text knowledge base; Step 3, deriving and expanding the combination of accident scenario elements based on the HyCon-Sg-Net algorithm; Step 4, strengthening and fine-tuning the Open-Sora model to achieve modal conversion from the text knowledge base to an accident video database, thereby constructing a multimodal accident scenario library. This invention can be used to test the performance of end-to-end autonomous vehicles in extreme accident scenarios, significantly improving the adaptability of autonomous driving algorithms to extreme accident scenarios.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving testing technology, specifically a method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing. Background Technology

[0002] In recent years, autonomous vehicles have developed rapidly, and the end-to-end autonomous driving paradigm has gradually become the mainstream of autonomous driving technology. It directly maps the input signals from perception sensors to the underlying control signals, which can reduce the cumulative error of traditional modular autonomous driving architectures. Because end-to-end autonomous vehicles directly input sensor information, compared with the testing tasks of traditional modular architectures, it places higher demands on scenario modalities. Traditional trajectory-based test scenario libraries can no longer achieve end-to-end autonomous driving performance testing.

[0003] The driving scenarios for autonomous vehicles are incredibly diverse. Among them, long-tail scenarios with low probability of occurrence but potential dangers, such as extreme weather, emergencies, and unexpected behaviors of road users, are the focus of test scenario library construction. However, daily data collection is inefficient and costly. Accident text reports are readily available, widely sourced, and abundant, making them an important source of long-tail accident scenarios. Building a text-video multimodal accident scenario library based on text reports can effectively reduce testing costs and significantly accelerate the testing efficiency of end-to-end autonomous vehicles.

[0004] Existing research struggles to efficiently and accurately extract and structure the storage of key accident information based on accident text reports, thus hindering the generation of high-quality, controllable text-based videos. There is an urgent need to break through the existing system architecture and leverage the precise query capabilities of a structured accident text knowledge base to generate targeted accident video data for efficient testing of the system under test. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for constructing a multimodal accident scenario database for end-to-end autonomous driving testing. It involves designing an efficient text information extraction algorithm to obtain key information from accident texts, designing an ontology-based knowledge graph storage architecture, constructing a multi-constraint-guided accident scenario derivation algorithm, and obtaining a large-scale accident text knowledge base. Furthermore, it designs an Open-Sora model-based enhancement and fine-tuning architecture, and achieves high-fidelity construction of an accident video database under specific extreme conditions through text information-oriented queries. This database can be used to test the performance of end-to-end autonomous vehicles in extreme accident scenarios, significantly improving the adaptability of autonomous driving algorithms to extreme accident scenarios.

[0006] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0007] This invention provides a method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing, comprising the following steps:

[0008] Step 1: Extract text information based on the MCAT-BiLSTM-CRF algorithm;

[0009] First, the accident text description is preprocessed, including three steps: text case conversion, redundant symbol deletion and replacement, and pronoun replacement. Then, the MCAT-BiLSTM-CRF algorithm is constructed to extract key information from the accident text description.

[0010] Step 2: Design the ontology architecture and use a knowledge graph to store accident scenario information to obtain an accident text knowledge base;

[0011] The accident scenario is described in the form of a combination of scenario elements. Then, an ontology architecture is designed, and the ontology information content of the accident scenario is divided into two parts: key information metadata and additional information metadata, forming an ontology concept layer. Then, based on the extracted key information of the accident scenario, the information is stored in the data application layer according to the correlation of entities in the ontology architecture. The ontology concept layer and the data application layer are mapped to each other to form a knowledge graph storage and management architecture.

[0012] Step 3: Based on the HyCon-Sg-Net algorithm, derive and extend the combination of accident scene elements;

[0013] The scene element combination sequence stored in the knowledge graph is encoded and mapped to meet the input format of the derivative algorithm. Then, the HyCon-Sg-Net derivative algorithm is constructed to extend the accident scene element combination. Based on step two, an extended accident scene text knowledge base is obtained.

[0014] Step 4: Enhance and fine-tune the Open-Sora model to achieve modal conversion from text knowledge base to accident video database, thereby constructing a multimodal accident scene library;

[0015] First, an Open-Sora fine-tuning dataset is constructed, and the text prompts in the fine-tuning dataset are refined as the basis for fine-tuning the Open-Sora model. Based on the accident scene text knowledge base constructed in step three, accident scene entries are queried in the text knowledge base according to the expected generation target, and an initial text prompt T is generated using a rule-based accident information combination method. init Then, using the ChatGPT-4o large language model, the initial prompt word T is... init Expand to the intensifying cue word T enhance The finely tuned Open-Sora model is guided to generate accident videos, realizing the modal conversion from accident text knowledge base to accident video data.

[0016] Furthermore, the specific method for step one is as follows:

[0017] 11) Preprocess the accident text description;

[0018] Accident text preprocessing is used to improve the standardization of accident text reports. The accident text preprocessing process includes text case conversion, deletion and replacement of redundant symbols, and pronoun replacement.

[0019] First, the accident text is converted to lowercase to standardize the writing format of the accident report. Then, redundant symbols are deleted and replaced. Redundant special symbols in the accident text are deleted, and special symbols with clear meanings are replaced with words. Finally, pronouns in the accident text are replaced, and pronouns in the text report are converted into actual targets.

[0020] 12) Construct the MCAT-BiLSTM-CRF algorithm for extracting key information from accident texts;

[0021] The MCAT-BiLSTM-CRF algorithm takes the preprocessed accident text as input, splits the token sequence and passes it through the input representation module, and then uses the BERT encoder, multi-scale semantic aggregation module, context enhancement module, BiLSTM network and structure-aware CRF decoder to process the accident information in sequence. By designing the training objectives of the algorithm, the key information in the accident text description is extracted.

[0022] First, the preprocessed accident text is segmented into words to obtain a token sequence T=[w1,w2,…,w...] of length n. n Each token is generated by a function. Convert to latent vector Embed the token in the input representation module. i Paragraph embeddings i Position embedding p i Add them together to obtain the input matrix of the BERT encoder. ;

[0023] The BERT encoder consists of L stacked Transformer blocks, which perform hierarchical feature extraction on the input matrix X and output multi-layered contextual representation information, as shown in Equation (1):

[0024] (1)

[0025] In the formula, For the first The hidden state at layer and position i; d is the dimension of the hidden state;

[0026] In the multi-scale semantic aggregation module, learnable cross-layer attention weights are introduced. The outputs from different layers are weighted and aggregated, as shown in equations (2) and (3):

[0027] (2)

[0028] (3)

[0029] In the formula, For cross-layer attention coefficients, satisfying m i The resulting multi-scale word vectors; u k Let be the scoring scalar for the k-th hidden state; For the first The scoring results corresponding to the layer;

[0030] In the context enhancement module, global semantic information is explicitly modeled, and the global vector c is calculated using equation (4):

[0031] (4)

[0032] Then, residual fusion is performed on the global vector c, as shown in equation (5):

[0033] (5)

[0034] In the formula, This represents the concatenation of vectors; FFN is a two-layer feedforward network. For global fusion semantics;

[0035] In obtaining global fusion semantics Then, Bi-LSTM is used to process the global fusion semantics, including both forward and reverse links, and the two links are concatenated at position t to obtain feature h. t As shown in equation (6):

[0036] (6)

[0037] In the formula, d h The dimension of the unidirectional latent vector;

[0038] Furthermore, in the structure-aware CRF decoder, the decoded label sequence y=[y1,…,y] is... n The display area is divided into two main categories: scene elements E T With additional information A I And in the CRF, the linear score is defined as shown in equation (7):

[0039] (7)

[0040] In the formula, It is a linear score function; This is the label weight vector; The transition matrix is ​​used, and the conditional probability is then expressed as equation (8):

[0041] (8)

[0042] In the formula, h 1:n It is the hidden state sequence from the 1st to the nth token; The predicted label sequence;

[0043] The MCAT-BiLSTM-CRF algorithm obtains the final predicted label for each token by solving the maximization sequence of equation (8), thereby acquiring the complete entity. The model training objective is shown in equation (9):

[0044] (9)

[0045] In the formula, y* represents the sequence of labeled tags.

[0046] Furthermore, the specific method for step two is as follows:

[0047] 21) Describe the combined elements of the accident scene;

[0048] The key information extracted from the text is decomposed into multi-dimensional scene elements to systematically represent the accident scene. The accident scene is often composed of the following scene elements: road type, environmental conditions, traffic participants and driving behavior. Different categories of scene elements are regarded as different dimensions describing the accident scene, and scene elements of each dimension are combined with each other. When describing the accident scene, the severity of the accident scene is integrated into the combination of accident scene elements to reflect the impact of different combinations of scene elements on the severity of the accident scene.

[0049] Multidimensional scene elements are described using element tuples, where R, E, P, B, and H represent road type, environmental conditions, traffic participants, driving behavior, and scene severity, respectively. The accident scene S is then represented as a five-tuple in equation (10):

[0050] (10)

[0051] In the formula, , , , , These represent the specific values ​​for road type, environmental conditions, traffic participants, driving behavior, and severity of the accident scenario.

[0052] 22) Design the ontological framework;

[0053] The ontological architecture corresponds to the ontological concept layer of the knowledge graph. In ontology, there are four concepts: class, entity, relation, and attribute. A class is a set of entities with similar attributes. An entity is the basic building block in an ontology. Relations are used to define the connection between classes or entities. Attributes are used to describe the concept of a class or entity.

[0054] The ontology architecture was constructed under the guidance of the Stanford seven-step method, which divides the accident text data into two parts: key information metadata and additional information metadata. The key information metadata is the combination of multi-dimensional elements of the accident scene, while the additional information metadata includes the accident description, accident time, accident location, and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to the entity association.

[0055] 23) Store accident information based on knowledge graphs;

[0056] Each entity in the key information metadata and supplementary information metadata corresponding to the ontological architecture is regarded as an entity node in the knowledge graph, and the corresponding entities are connected according to the attribute relationships between the information to represent a complete accident scenario. For each key information metadata, a key information relationship triplet is added to the knowledge graph, as shown in Equation (11). For each supplementary information metadata, a supplementary information relationship triplet is added, as shown in Equation (12).

[0057] (11)

[0058] (12)

[0059] In the formula, ScenarioID i is the scene root node, used to represent the relevant information of the i-th accident scene; hasKey is the connection between the scene root node and key information metadata; hasAdditional is the connection between the scene root node and additional information metadata; k is the key information metadata; a is the additional information metadata;

[0060] In the constructed knowledge graph, all accident scenario information is organized into a multi-level knowledge graph structure. Nodes of different categories are aggregated through the implicit relationship of "belonging to the same scenario" to form a complete description of the accident scenario. Based on this, Neo4j software is used to realize the storage of accident information based on the knowledge graph.

[0061] Furthermore, the specific method for step three is as follows:

[0062] 31) Encode and map the combination sequence of scene elements;

[0063] The semantically represented combination of scene elements is converted into a numerical encoding form that can be processed by the algorithm. For each multi-dimensional combination of elements S=(r,e,p,b,h) under the accident scenario, the scene elements of different dimensions are mapped to the corresponding numerical discrete codes.

[0064] Let L r L e L p L b L h These represent the sizes of scene element sets in different dimensions, and define the encoding function f for each dimension of scene elements. R :r→1,…,L r f E :e→1,…,L e f P :p→1,…,L p f B :b→1,…,L b f H :h→1,…,L h By using an encoding function to map the scene elements in each set to their corresponding numbers, the scene element combination S=(r,e,p,b,h) can be represented as a fixed-length discrete encoding sequence, as shown in equation (13):

[0065] (13)

[0066] In the formula, enc(S) is a sequence composed of discrete codes;

[0067] The meaning of the encoding at each position in the encoded sequence is determined by the dimension in which it is located; after discrete encoding mapping, the original set of scene element combinations S orig Converted into a set of encoded sequences S enc ;

[0068] 32) Construct the HyCon-Sg-Net algorithm for deriving scene element combination sequences;

[0069] The HyCon-Sg-Net algorithm is constructed to extend the combination of accident scene elements, integrating the advantages of diffusion models and generative adversarial networks, and combining multiple guidance mechanisms; the HyCon-Sg-Net algorithm uses an encoder to process the input scene element encoding S enc Then, through a forward diffusion process, a conditional generation-reverse denoising process, and with the help of multiple guidance modules, generated sample pairs are obtained. These samples are then input into the discriminator along with real sample pairs for discrimination. In conjunction with the training objective of the algorithm, a high-confidence expansion of the accident text knowledge base is achieved.

[0070] In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and the log-variance vector logσ of the latent variables. 2 Furthermore, the reparameterization technique is employed to sample the latent vector x in the Gaussian distribution. T As shown in equation (14):

[0071] (14)

[0072] In the forward diffusion process, the diffusion probability model is used to determine the sample x0=x T x is generated by gradually adding noise within time step T. t Let β t The noise injection process at each time step is shown in Equation (15), which represents the variance increment:

[0073] (15)

[0074] In the formula, The obtained x t That is, the input vector for the reverse denoising process, q(x) t |x t-1 ) represents the transition probability distribution during the forward diffusion process;

[0075] In the conditional generation-inverse denoising process, the goal of inverse denoising is to transform the input x under condition c. t Gradually restored to x0, the generator prediction noise is Where z is a random latent variable, and x is restored t-1 The mean is expressed as shown in equation (16):

[0076] (16)

[0077] In the formula, μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in Equation (17):

[0078] (17)

[0079] In the formula, This represents the probability distribution of the backsampling process; The variance is the learnable variance; N is a Gaussian distribution.

[0080] Discriminator D φ Using real samples With generated sample pairs As input, a non-saturated adversarial target is adopted, as shown in equations (18) and (19):

[0081] (18)

[0082] (19)

[0083] In the formula, η is the noise sampled from the Gaussian distribution, i.e. ; q represents the data distribution of the real sample; G θ The data distribution for generating samples for the generator;

[0084] In the multi-guidance module, with probability P drop The random null condition c is used for joint prediction during inference based on equation (20) to coordinate the controllability and diversity of target generation:

[0085] (20)

[0086] In the formula, This is the result generated under unconditional c; This represents the generated result under condition c; λ1 is the weight adjustment factor, with a value between 0 and 1.

[0087] In the multi-guidance module, unreasonable combinations of elements are excluded by injecting prior violation conditions; the rule evaluation function r(x0,c) is defined. g ), used to determine whether sample x0 meets the violation condition c. g Thus, the constrained guided loss is obtained as shown in equation (21):

[0088] (twenty one)

[0089] In a multi-boot module, for the same x T Repeat sampling M times, and evaluate the output set using equation (22). The variance, if σ 2 >δ, where δ is the set threshold, indicates that the fluctuation between samples is too large, the model refuses to output and resamples;

[0090] (twenty two)

[0091] In the formula, d is the dimension of the generated sample; This represents the value of the i-th dimension in the m-th sampling result;

[0092] The overall training loss of the HyCon-Sg-Net algorithm is shown in Equation (23):

[0093] (twenty three)

[0094] In the formula, L total Total loss; L diff L represents the noise prediction error of the diffusion probability model DDPM; G L DTo combat the losses, L guide To constrain the guided loss, λ G , λ D , λ guide is the weighting coefficient for the loss.

[0095] Furthermore, the specific method for step four is as follows:

[0096] 41) Construct an Open-Sora model fine-tuning dataset, refine the text prompts in the fine-tuning dataset, and fine-tune the Open-Sora model;

[0097] We collected real traffic accident videos from open-source datasets and online platforms; we manually wrote rough text descriptions for each accident video, which accurately summarized the key scene elements and the sequence of events in the video.

[0098] Building upon the labeled prompts, the Video-LLaMA 2 model, a large-scale model for multimodal data understanding, is introduced to refine and enhance the coarsely labeled prompts. Accident videos and manually labeled coarse prompts are input into the model, which leverages its powerful visual semantic understanding capabilities to generate more detailed and accurate text descriptions. This supplements missing environmental details and traffic participant actions, and corrects descriptions that do not match the video footage, resulting in a finely adjusted and enhanced prompt T. ref1 ;

[0099] The enhanced text description and the original accident video constitute a high-quality training pair (T). ref1 Based on this dataset, the Open-Sora model parameters θ are fine-tuned and trained. The model is updated by maximizing the log-likelihood of the real video under the text condition, as shown in Equation (24):

[0100] (twenty four)

[0101] In the formula, D ft For fine-tuning the video-text dataset; p θ (V|T) represents the probability distribution of the model generating video V from text T;

[0102] 42) Refinement of rule combination prompts and generation of accident video guidance;

[0103] Using a knowledge graph-based accident scenario text knowledge base, accident scenario entries are queried in the text knowledge base according to the expected generation target. Then, an initial text prompt word T is generated using a rule-based accident information combination method. init The initial prompt is simply composed of various entity nodes under a root node in the knowledge graph. Then, the ChatGPT-4o large language model is used to process the initial prompt word T. initExpanded to a more detailed and refined intensifying cue word T enhance , and thus T enhance By inputting the finely tuned Open-Sora model, accident videos that match the refined prompt word descriptions are generated. This completes the transformation from an accident text knowledge base to an accident video database, resulting in a multimodal accident scenario database that can be used for targeted testing of end-to-end autonomous vehicles.

[0104] The beneficial effects of this invention are as follows:

[0105] The multimodal accident scenario library construction method for end-to-end autonomous vehicle testing provided by this invention can generate a multimodal accident scenario library including an accident text knowledge base and an accident video database based on easily accessible accident text report information. By performing targeted queries in the accident text knowledge base and generating refined text prompts, it can guide the model to generate extreme accident scenario videos consistent with the text descriptions. This can fully verify the performance of end-to-end autonomous vehicles in extreme accident scenarios and test the robustness of end-to-end autonomous driving perception algorithms in extreme accident scenarios. Attached Figure Description

[0106] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a flowchart of the present invention;

[0108] Figure 2 Diagram of the MCAT-BiLSTM-CRF algorithm architecture;

[0109] Figure 3 This is a diagram of the ontology concept layer architecture.

[0110] Figure 4 A mapping diagram between the ontology concept layer and the data application layer;

[0111] Figure 5 Diagram of the HyCon-Sg-Net algorithm architecture;

[0112] Figure 6 Refine the architecture diagram for the Open-Sora model;

[0113] Figure 7 This is a case study of the failure of an end-to-end autonomous vehicle target detection algorithm in an extreme accident scenario. Detailed Implementation

[0114] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0115] Example 1

[0116] See Figure 1 This embodiment provides a method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing, including the following steps:

[0117] Step 1: Extract text information based on the MCAT-BiLSTM-CRF algorithm;

[0118] First, the accident text description is preprocessed, including three steps: text case conversion, redundant symbol removal and replacement, and pronoun replacement. Then, the MCAT-BiLSTM-CRF algorithm is constructed to extract key information from the accident text description, as follows:

[0119] 11) Preprocess the accident text description;

[0120] Accident text preprocessing is used to improve the standardization of accident text reports and reduce the inaccuracy of key information extraction due to differences in text writing style. The process includes three steps: text case conversion, redundant symbol deletion and replacement, and pronoun replacement.

[0121] First, the accident text is case-converted to standardize the writing format of the accident report, converting uppercase and lowercase characters to lowercase. Then, redundant symbols are removed and replaced. Redundant special symbols such as " / " and "*" are removed to avoid affecting the extraction of key information. Special symbols with clear meanings are replaced with words, such as "percentages" instead of "%". Finally, pronouns are replaced in the accident text, converting pronouns such as "it" and "the" into actual referents to ensure the completeness of the text report information and avoid difficulties in algorithm recognition.

[0122] 12) Construct the MCAT-BiLSTM-CRF algorithm for extracting key information from accident texts;

[0123] See Figure 2 The MCAT-BiLSTM-CRF algorithm consists of an input representation module, a BERT encoder, a multi-scale semantic aggregation module, a context enhancement module, a BiLSTM network, and a structure-aware CRF decoder.

[0124] The input representation module is used for text word segmentation processing. It obtains the input matrix of the BERT encoder by adding multi-dimensional embedding vectors.

[0125] The BERT encoder is used for hierarchical feature extraction from the input matrix and outputs multi-layered contextual representation information.

[0126] The multi-scale semantic aggregation module is used to perform weighted aggregation of the outputs from different layers;

[0127] The context enhancement module is used for display modeling of global semantic information, strengthens the ability to process cross-sentence dependency features, and improves the problem of ambiguous entity boundaries;

[0128] BiLSTM networks are used to process global semantics and further characterize text temporal dependencies and entity boundaries.

[0129] A structure-aware CRF decoder is used to decode the predicted label sequence.

[0130] The MCAT-BiLSTM-CRF algorithm takes the preprocessed accident text as input, splits the token sequence and passes it through the input representation module, and then uses the BERT encoder, multi-scale semantic aggregation module, context enhancement module, BiLSTM network and structure-aware CRF decoder to process the accident information in sequence. By designing the training objectives of the algorithm, key information in the accident text description can be extracted.

[0131] First, the preprocessed accident text is segmented into words to obtain a token sequence T=[w1,w2,…,w...] of length n. n Each token is generated by a function. Convert to latent vector Embed the token in the input representation module. i Paragraph embeddings i Position embedding p i Add them together to obtain the input matrix of the BERT encoder. ;

[0132] The BERT encoder consists of L stacked Transformer blocks, which extract hierarchical features from the input matrix X and output multi-layered contextual representation information, as shown in Equation (1):

[0133] (1)

[0134] In the formula, For the first The hidden state at layer and position i; d is the dimension of the hidden state;

[0135] In the multi-scale semantic aggregation module, learnable cross-layer attention weights are introduced. The outputs from different layers are weighted and aggregated, as shown in equations (2) and (3):

[0136] (2)

[0137] (3)

[0138] In the formula, For cross-layer attention coefficients, satisfying m i The resulting multi-scale word vectors; u k Let be the scoring scalar for the k-th hidden state; For the first The scoring results corresponding to the layer;

[0139] In the context enhancement module, global semantic information is explicitly modeled, and the global vector c is calculated using equation (4):

[0140] (4)

[0141] Then, residual fusion is performed on the global vector c, as shown in equation (5):

[0142] (5)

[0143] In the formula, This represents the concatenation of vectors; FFN is a two-layer feedforward network. For global semantic fusion, this design effectively enhances the ability to handle cross-sentence dependency features, improves the problem of ambiguous entity boundaries, and is suitable for long accident text descriptions.

[0144] In obtaining global fusion semantics Then, Bi-LSTM is used to process the global fusion semantics, which includes two links, forward and reverse, and the two links are concatenated at position t to obtain feature h. t As shown in equation (6):

[0145] (6)

[0146] In the formula, d h The dimension of the unidirectional latent vector;

[0147] Furthermore, in the structure-aware CRF decoder, this paper decodes the label sequence y=[y1,…,y...]. n The display area is divided into two main categories: scene elements E T With additional information A IAnd in the CRF, the linear score is defined as shown in equation (7):

[0148] (7)

[0149] In the formula, It is a linear score function; This is the label weight vector; The transition matrix is ​​used, and the conditional probability is then expressed as equation (8):

[0150] (8)

[0151] In the formula, h 1:n It is the hidden state sequence from the 1st to the nth token; The predicted label sequence;

[0152] The MCAT-BiLSTM-CRF algorithm obtains the final predicted label for each token by solving the maximization sequence of equation (8), thereby obtaining the complete entity. The model training objective is shown in equation (9):

[0153] (9)

[0154] In the formula, y* represents the sequence of labeled tags;

[0155] Step 2: Design the ontology architecture and use a knowledge graph to store accident scenario information to obtain an accident text knowledge base;

[0156] The accident scenario is described as a combination of scenario elements. Then, an ontology architecture is designed, dividing the ontology information of the accident scenario into two parts: key information metadata and supplementary information metadata, forming an ontology concept layer. Subsequently, based on the extracted key information of the accident scenario, the information is stored in the data application layer according to the relationships between entities in the ontology architecture. The ontology concept layer and the data application layer are mutually mapped to form a knowledge graph storage and management architecture, as detailed below:

[0157] 21) Describe the combined elements of the accident scene;

[0158] The key information extracted from the text is decomposed into multi-dimensional scene elements to systematically represent accident scenarios. Accident scenarios are often composed of the following scene elements: road type, environmental conditions, traffic participants, and driving behavior. This invention considers different categories of scene elements as different dimensions describing accident scenarios, and these scene elements are combined with each other. It should be noted that the severity of the accident is not included in the scene element set E. TIt is a subset of the accident scene, but it represents the accident hazards caused by the combination of scene elements. When describing the accident scene, the severity of the accident scene is incorporated into the combination of accident scene elements to reflect the impact of different combinations of scene elements on the severity of the accident scene.

[0159] Using element tuples to describe multidimensional scene elements, R, E, P, B, and H represent road type, environmental conditions, traffic participants, driving behavior, and scene severity, respectively. Thus, the accident scene S can be represented as a five-tuple form as shown in equation (10):

[0160] (10)

[0161] In the formula, , , , , These represent the specific values ​​for road type, environmental conditions, traffic participants, driving behavior, and severity of a given accident scenario.

[0162] 22) Design the ontological framework;

[0163] The ontological architecture corresponds to the ontological concept layer of a knowledge graph. For the architecture of the ontological concept layer, please refer to [link / reference needed]. Figure 3 In ontology, there are four concepts: class, entity, relation, and attribute. A class is a collection of entities with similar attributes. An entity is the basic building block of an ontology. Relations are used to define the connection between classes or entities. Attributes are used to describe the concept of a class or entity.

[0164] The ontology architecture in this invention is constructed under the guidance of the Stanford seven-step method, dividing accident text data into two parts: key information metadata and supplementary information metadata. Key information metadata consists of a multi-dimensional combination of accident scene elements, while supplementary information metadata includes additional information such as accident description, accident time, accident location, and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to entity relationships. The mapping relationship between the ontology concept layer and the data application layer is described in [reference needed]. Figure 4 ;

[0165] 23) Store accident information based on knowledge graphs;

[0166] This invention treats each entity in the key information metadata and supplementary information metadata corresponding to the ontological architecture as an entity node in a knowledge graph, and connects the corresponding entities according to the attribute relationships between the information to represent a complete accident scenario. Specifically, for each key information metadata, a key information relationship triplet is added to the knowledge graph, as shown in Equation (11), and for each supplementary information metadata, an supplementary information relationship triplet is added, as shown in Equation (12).

[0167] (11)

[0168] (12)

[0169] In the formula, ScenarioID 为 The scene root node is used to represent the relevant information of the i-th accident scene. "hasKey" is used to connect the scene root node with key information metadata, and "hasAdditional" is used to connect the scene root node with additional information metadata. k is key information metadata and a is additional information metadata.

[0170] In the constructed knowledge graph, all accident scene information is organized into a multi-level knowledge graph structure. Nodes of different categories are aggregated through the implicit relationship of "belonging to the same scene" to form a complete description of the accident scene. Based on this, the present invention uses Neo4j software to realize accident information storage based on knowledge graph, introduces semantic Web rule language into ontology, which helps to realize knowledge reasoning based on ontology and mine hidden information in knowledge graph.

[0171] Step 3: Based on the HyCon-Sg-Net algorithm, derive and extend the combination of accident scene elements;

[0172] The sequence of scene element combinations stored in the knowledge graph is encoded and mapped to meet the input format of the derivative algorithm. Then, the HyCon-Sg-Net derivative algorithm is constructed to extend the combination of accident scene elements. Based on step two, the extended accident scene text knowledge base is obtained, as follows:

[0173] 31) Encode and map the combination sequence of scene elements;

[0174] The semantically represented combination of scene elements is converted into a numerical encoding form that can be processed by the algorithm. Specifically, for each multi-dimensional combination of elements S=(r,e,p,b,h) under an accident scenario, the scene elements of different dimensions are mapped to the corresponding numerical discrete codes.

[0175] Let L r L e L p Lb L h These represent the sizes of scene element sets in different dimensions, and define the encoding function f for each dimension of scene elements. R :r→1,…,L r f E :e→1,…,L e f P :p→1,…,L p f B :b→1,…,L b f H :h→1,…,L h By using an encoding function to map the scene elements in each set to their corresponding numbers, the scene element combination S=(r,e,p,b,h) can be represented as a fixed-length discrete encoding sequence, as shown in equation (13):

[0176] (13)

[0177] In the formula, enc(S) is a sequence composed of discrete codes;

[0178] The meaning of each position in the encoding sequence is determined by its dimension. For example, the first position in the encoding sequence always represents the road type, and the second position represents the environmental conditions. The encoding sequence essentially preserves the semantic structure of scene elements. After discrete encoding mapping, the original set of scene element combinations S... orig Converted into a set of encoded sequences S enc ;

[0179] In addition, to ensure the completeness of the derived accident scene information, additional information metadata such as accident time and accident location are randomly extracted from the original scene information database and combined. This is only used to maintain the consistency of the overall structure when storing the knowledge graph, and there is no need to focus on the accuracy of such information. It does not affect the generation of accident scene video data and the testing process of autonomous vehicles.

[0180] 32) Construct the HyCon-Sg-Net algorithm for deriving scene element combination sequences;

[0181] See Figure 5 This invention constructs the HyCon-Sg-Net (Hybrid Constrained Scenario-generation Network) algorithm to extend the combination of accident scene elements. It integrates the advantages of diffusion models and generative adversarial networks and combines multiple guidance mechanisms to effectively ensure the physical compliance and credibility of accident scene generation. The HyCon-Sg-Net algorithm uses an encoder to process the input scene element encoding S. encThen, through a forward diffusion process, a conditional generation-reverse denoising process, and with the help of multiple guidance modules, generated sample pairs are obtained. These samples are then input into the discriminator along with real sample pairs for discrimination. By using the above process and in conjunction with the training objectives of the algorithm, a high-confidence expansion of the accident text knowledge base can be achieved.

[0182] In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and the log-variance vector logσ of the latent variables. 2 Furthermore, the reparameterization technique is employed to sample the latent vector x in the Gaussian distribution. T As shown in equation (14):

[0183] (14)

[0184] In the forward diffusion process, the Denoising Diffusion Probabilistic Models (DDPM) are used to determine the sample x0=x T x is generated by gradually adding noise within time step T. t Let β t The noise injection process at each time step is shown in Equation (15), which represents the variance increment:

[0185] (15)

[0186] In the formula, The obtained x t This is the input vector for the reverse denoising process; q(x) t |x t-1 ) represents the transition probability distribution during the forward diffusion process;

[0187] In the conditional generation-inverse denoising process, the goal of inverse denoising is to transform the input x under condition c. t Gradually restored to x0, the generator prediction noise is Where z is a random latent variable, and x is restored t-1 The mean is expressed as shown in equation (16):

[0188] (16)

[0189] In the formula, μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in Equation (17):

[0190] (17)

[0191] In the formula, This represents the probability distribution of the backsampling process; The variance is the learnable variance; N is a Gaussian distribution.

[0192] Discriminator D φ Using real samples With generated sample pairs As input, to avoid gradient vanishing, a non-saturated adversarial target is adopted, as shown in equations (18) and (19):

[0193] (18)

[0194] (19)

[0195] In the formula, η is the noise sampled from the Gaussian distribution, i.e. ; q represents the data distribution of the real sample; G θ The data distribution for generating samples for the generator;

[0196] In the multi-guidance module, with probability P drop The random null condition c is used for joint prediction during inference based on equation (20) to coordinate the controllability and diversity of target generation:

[0197] (20)

[0198] In the formula, This is the result generated under unconditional c; This represents the generated result under condition c; λ1 is the weight adjustment factor, with a value between 0 and 1.

[0199] In the multi-guidance module of the HyCon-Sg-Net algorithm, unreasonable combinations of elements such as "heavy fog + strong light" or "dark night + strong sunlight" are excluded by injecting prior violation conditions. The rule evaluation function r(x0,c) is defined. g ), used to determine whether sample x0 meets the violation condition c. g Thus, the constrained guided loss is obtained as shown in equation (21):

[0200] (twenty one)

[0201] In a multi-boot module, for the same x T Repeat sampling M times, and evaluate the output set using equation (22). The variance, if σ 2 >δ (δ is the set threshold) indicates that the fluctuation between samples is too large. The model refuses to output and resamples to improve the reliability of the output.

[0202] (twenty two)

[0203] In the formula, d represents the dimension of the generated sample. This represents the value of the i-th dimension in the m-th sampling result;

[0204] The overall training loss of the HyCon-Sg-Net algorithm is shown in Equation (23):

[0205] (twenty three)

[0206] In the formula, L total L represents the total loss. diff L represents the noise prediction error of the diffusion probability model DDPM. G L D To combat the losses, L guide To constrain the guided loss, λ G , λ D , λ guide is the weighting coefficient for the loss.

[0207] Step 4: Enhance and fine-tune the Open-Sora model to achieve modal conversion from text knowledge base to accident video database, thereby constructing a multimodal accident scene library;

[0208] See Figure 6 First, an Open-Sora fine-tuning dataset is constructed, and the text prompts in the fine-tuning dataset are refined as the basis for fine-tuning the Open-Sora model. Based on the accident scene text knowledge base constructed in step three, accident scene entries are queried in the text knowledge base according to the expected generation target, and the initial text prompt T is generated using a rule-based accident information combination method. init Then, using the ChatGPT-4o large language model, the initial prompt word T is... init Expand to the intensifying cue word T enhance The finely tuned Open-Sora model is guided to generate accident videos, realizing modal conversion from accident text knowledge base to accident video data, as detailed below:

[0209] 41) Construct an Open-Sora model fine-tuning dataset, refine the text prompts in the fine-tuning dataset, and fine-tune the Open-Sora model;

[0210] This invention collects real traffic accident videos from open-source datasets and online platforms. After manual screening to ensure clear image quality, the quality of the dataset is fine-tuned. Next, a rough text description prompt is written for each accident video by human intervention. The rough description needs to accurately summarize the key scene elements and accident sequence process contained in the video.

[0211] Building upon manually annotated warning words, the Video-LLaMA 2 model, a large-scale model for multimodal data understanding, is introduced to refine and enhance the coarsely annotated warning words. Accident videos and manually annotated coarse warning words are input into the model, which leverages its powerful visual semantic understanding capabilities to generate more detailed and accurate text descriptions. This supplements environmental details and traffic participant actions missed by manual annotations and corrects descriptions that do not match the visuals, resulting in a finely adjusted and enhanced warning word T. ref1 ;

[0212] The enhanced text description and the original accident video constitute a high-quality training pair (T). ref1 Based on this dataset, this paper fine-tunes the Open-Sora model parameters θ and updates the model by maximizing the log-likelihood of the real video under text conditions, as shown in Equation (24):

[0213] (twenty four)

[0214] In the formula, D ft This represents a video-text dataset for fine-tuning, p θ (V|T) represents the probability distribution of the model generating video V from text T. The optimization process is to minimize the difference between the generated video and the real video, so that the model can gradually learn to generate realistic accident videos under the prompts of accident text description.

[0215] 42) Refinement of rule combination prompts and generation of accident video guidance;

[0216] Using a knowledge graph-based accident scenario text knowledge base, accident scenario entries are queried in the text knowledge base according to the expected generation target. Then, an initial text prompt word T is generated using a rule-based accident information combination method. init The initial prompt is simply composed of various entity nodes under a root node in the knowledge graph. Then, the ChatGPT-4o large language model is used to process the initial prompt word T. init Expanded to a more detailed and refined intensifying cue word T enhance , and thus T enhance By inputting the finely tuned Open-Sora model, accident videos that match the refined prompt description are generated. This completes the transformation from an accident text knowledge base to an accident video database, resulting in a multimodal accident scenario database that can be used for targeted testing of end-to-end autonomous vehicles.

[0217] Example 2

[0218] A multimodal accident scenario database was constructed using the method provided in Example 1 to test the robustness of autonomous vehicle perception algorithms under extreme accident scenarios. Accident scenarios are often sudden and represent typical extreme scenario types. Successful identification of potentially dangerous vehicles within a certain distance by the perception algorithm carried by the main vehicle is a necessary prerequisite for the main vehicle to avoid danger. If the main vehicle's perception algorithm continuously fails, it indicates that the algorithm has certain performance defects, which can easily lead to accidents. The extreme accident scenarios in this invention refer to scenarios that pose significant challenges to the robust identification of perception algorithms, including highly challenging scenario elements such as darkness, strong glare, and heavy fog.

[0219] This invention leverages the semantic query capabilities of an accident text knowledge base to generate 100 accident scene videos under extreme conditions as detection dataset 1. Specifically, extreme accident scene cases are selectively queried and extracted from the accident text knowledge base stored in the form of a knowledge graph, generating prompts such as "The main vehicle is driving in the dark without light, a black vehicle in front slows down, and the main vehicle rear-ends it." This prompt guides the fine-tuned Open-Sora model to generate accident videos. 100 accident videos are randomly selected from the accident dataset used for model fine-tuning as detection dataset 2 for algorithm performance comparison. The YOLOv8 and DeepSORT algorithms are used to identify and track dangerous vehicles in accident scenes. The detection success rates of the main vehicle perception algorithm under different detection datasets are shown in Table 1.

[0220] Table 1. Performance of the perceptron algorithm on different datasets

[0221] Dataset Number of successful tests Detection success rate Dataset 1 36 / 100 0.36 Dataset 2 84 / 100 0.84

[0222] As shown in Table 1, the targeted querying of extreme accident cases in the text knowledge base to generate accident video data under extreme conditions poses a significant challenge to the performance of vehicle perception algorithms. Compared with publicly available accident video datasets, the detection success rate is reduced by up to 48%, indicating that the extreme accident scene videos generated by this invention can reveal the functional limitations of perception algorithms and improve the ability of end-to-end autonomous vehicles to perceive dangerous vehicles in extreme accident scenarios.

[0223] Please refer to case studies of end-to-end autonomous vehicle target detection algorithm failure in extreme accident scenarios. Figure 7In this case of perception failure, a white vehicle was approaching the main vehicle from the right. Because it was very similar to the white background of the snowy weather, the perception algorithm failed to identify the white vehicle as it approached the main vehicle at high speed (frames 3, 22, and 41). In frames 3 and 22, the main vehicle still had a chance to take action to avoid the vehicle and was within the safe operating range. However, in frame 41, the distance between the vehicles was too close, and the situation was highly dangerous. Around frame 57, the main vehicle's perception algorithm successfully identified the white vehicle, but by then the white vehicle was too close to the main vehicle, and the main vehicle did not have time to avoid it. This scenario reveals the detection defect of the perception algorithm for white vehicles under snowy conditions.

[0224] In summary, the multimodal accident scenario library construction method for end-to-end autonomous vehicle testing provided by this invention can generate a multimodal accident scenario library including an accident text knowledge base and an accident video database based on easily accessible and widely available accident text reports. By performing targeted queries in the text knowledge base and generating refined text prompts, the model is guided to generate extreme accident scenario videos consistent with the text descriptions. This can fully verify the performance of end-to-end autonomous vehicles in extreme accident scenarios and test the robustness of end-to-end autonomous driving perception algorithms in extreme accident scenarios.

[0225] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing, characterized in that, Includes the following steps: Step 1: Extract text information based on the MCAT-BiLSTM-CRF algorithm; First, the accident text description is preprocessed, including three steps: text case conversion, redundant symbol deletion and replacement, and pronoun replacement. Then, the MCAT-BiLSTM-CRF algorithm is constructed to extract key information from the accident text description. Step 2: Design the ontology architecture and use a knowledge graph to store accident scenario information to obtain an accident text knowledge base; The accident scenario is described in the form of a combination of scenario elements. Then, an ontology architecture is designed, and the ontology information content of the accident scenario is divided into two parts: key information metadata and additional information metadata, forming an ontology concept layer. Then, based on the extracted key information of the accident scenario, the information is stored in the data application layer according to the correlation of entities in the ontology architecture. The ontology concept layer and the data application layer are mapped to each other to form a knowledge graph storage and management architecture. Step 3: Based on the HyCon-Sg-Net algorithm, derive and extend the combination of accident scene elements; The scene element combination sequence stored in the knowledge graph is encoded and mapped to meet the input format of the derivative algorithm. Then, the HyCon-Sg-Net derivative algorithm is constructed to extend the accident scene element combination. Based on step two, an extended accident scene text knowledge base is obtained. Step 4: Enhance and fine-tune the Open-Sora model to achieve modal conversion from text knowledge base to accident video database, thereby constructing a multimodal accident scene library; First, an Open-Sora fine-tuning dataset is constructed, and the text prompts in the fine-tuning dataset are refined as the basis for fine-tuning the Open-Sora model. Based on the accident scene text knowledge base constructed in step three, accident scene entries are queried in the text knowledge base according to the expected generation target, and an initial text prompt T is generated using a rule-based accident information combination method. init Then, using the ChatGPT-4o large language model, the initial prompt word T is... init Expand to the intensifying cue word T enhance The finely tuned Open-Sora model is guided to generate accident videos, realizing the modal conversion from accident text knowledge base to accident video data.

2. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that, The specific method for step one is as follows: 11) Preprocess the accident text description; Accident text preprocessing is used to improve the standardization of accident text reports. The accident text preprocessing process includes text case conversion, deletion and replacement of redundant symbols, and pronoun replacement. First, the accident text is converted to lowercase to standardize the writing format of the accident report. Then, redundant symbols are deleted and replaced. Redundant special symbols in the accident text are deleted, and special symbols with clear meanings are replaced with words. Finally, pronouns in the accident text are replaced, and pronouns in the text report are converted into actual targets. 12) Construct the MCAT-BiLSTM-CRF algorithm for extracting key information from accident texts; The MCAT-BiLSTM-CRF algorithm takes the preprocessed accident text as input, splits the token sequence and passes it through the input representation module, and then uses the BERT encoder, multi-scale semantic aggregation module, context enhancement module, BiLSTM network and structure-aware CRF decoder to process the accident information in sequence. By designing the training objectives of the algorithm, the key information in the accident text description is extracted. First, the preprocessed accident text is segmented into words to obtain a token sequence T=[w1,w2,…,w...] of length n. n Each token is generated by a function. Convert to latent vector Embed the token in the input representation module. i Paragraph embeddings i Position embedding p i Add them together to obtain the input matrix of the BERT encoder. ; The BERT encoder consists of L stacked Transformer blocks, which perform hierarchical feature extraction on the input matrix X and output multi-layered contextual representation information, as shown in Equation (1): (1) In the formula, For the first The hidden state at layer and position i; d is the dimension of the hidden state; In the multi-scale semantic aggregation module, learnable cross-layer attention weights are introduced. The outputs from different layers are weighted and aggregated, as shown in equations (2) and (3): (2) (3) In the formula, For cross-layer attention coefficients, satisfying m i The resulting multi-scale word vectors; u k Let be the scoring scalar for the k-th hidden state; For the first The scoring results corresponding to the layer; In the context enhancement module, global semantic information is explicitly modeled, and the global vector c is calculated using equation (4): (4) Then, residual fusion is performed on the global vector c, as shown in equation (5): (5) In the formula, This represents the concatenation of vectors; FFN is a two-layer feedforward network. For global fusion semantics; In obtaining global fusion semantics Then, Bi-LSTM is used to process the global fusion semantics, including both forward and reverse links, and the two links are concatenated at position t to obtain feature h. t As shown in equation (6): (6) In the formula, d h The dimension of the unidirectional latent vector; Furthermore, in the structure-aware CRF decoder, the decoded label sequence y=[y1,…,y] is... n The display area is divided into two main categories: scene elements E T With additional information A I And in the CRF, the linear score is defined as shown in equation (7): (7) In the formula, It is a linear score function; This is the label weight vector; The transition matrix is ​​used, and the conditional probability is then expressed as equation (8): (8) In the formula, h 1:n It is the hidden state sequence from the 1st to the nth token; The predicted label sequence; The MCAT-BiLSTM-CRF algorithm obtains the final predicted label for each token by solving the maximization sequence of equation (8), thereby acquiring the complete entity. The model training objective is shown in equation (9): (9) In the formula, y* represents the sequence of labeled tags.

3. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that, The specific method for step two is as follows: 21) Describe the combined elements of the accident scene; The key information extracted from the text is decomposed into multi-dimensional scene elements to systematically represent the accident scene. The accident scene is often composed of the following scene elements: road type, environmental conditions, traffic participants and driving behavior. Different categories of scene elements are regarded as different dimensions describing the accident scene, and scene elements of each dimension are combined with each other. When describing the accident scene, the severity of the accident scene is integrated into the combination of accident scene elements to reflect the impact of different combinations of scene elements on the severity of the accident scene. Multidimensional scene elements are described using element tuples, where R, E, P, B, and H represent road type, environmental conditions, traffic participants, driving behavior, and scene severity, respectively. The accident scene S is then represented as a five-tuple in equation (10): (10) In the formula, , , , , These represent the specific values ​​for road type, environmental conditions, traffic participants, driving behavior, and severity of the accident scenario. 22) Design the ontological framework; The ontological architecture corresponds to the ontological concept layer of the knowledge graph. In ontology, there are four concepts: class, entity, relation, and attribute. A class is a set of entities with similar attributes. An entity is the basic building block in an ontology. Relations are used to define the connection between classes or entities. Attributes are used to describe the concept of a class or entity. The ontology module was constructed under the guidance of the Stanford seven-step method, which divides the accident text data into two parts: key information metadata and additional information metadata. The key information metadata is a combination of multi-dimensional elements of the accident scene, while the additional information metadata includes the accident description, accident time, accident location, and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to entity relationships. 23) Store accident information based on knowledge graphs; Each entity in the key information metadata and supplementary information metadata corresponding to the ontological architecture is regarded as an entity node in the knowledge graph, and the corresponding entities are connected according to the attribute relationships between the information to represent a complete accident scenario. For each key information metadata, a key information relationship triplet is added to the knowledge graph, as shown in Equation (11). For each supplementary information metadata, a supplementary information relationship triplet is added, as shown in Equation (12). (11) (12) In the formula, ScenarioID i is the scene root node, used to represent the relevant information of the i-th accident scene; hasKey is the connection between the scene root node and key information metadata; hasAdditional is the connection between the scene root node and additional information metadata; k is the key information metadata; a is the additional information metadata; In the constructed knowledge graph, all accident scenario information is organized into a multi-level knowledge graph structure. Nodes of different categories are aggregated through the implicit relationship of "belonging to the same scenario" to form a complete description of the accident scenario. Based on this, Neo4j software is used to realize the storage of accident information based on the knowledge graph.

4. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that, The specific method for step three is as follows: 31) Encode and map the combination sequence of scene elements; The semantically represented combination of scene elements is converted into a numerical encoding form that can be processed by the algorithm. For each multi-dimensional combination of elements S=(r,e,p,b,h) under the accident scenario, the scene elements of different dimensions are mapped to the corresponding numerical discrete codes. Let L r L e L p L b L h These represent the sizes of scene element sets in different dimensions, and define the encoding function f for each dimension of scene elements. R :r→1,…,L r f E :e→1,…,L e f P :p→1,…,L p f B :b→1,…,L b f H :h→1,…,L h By using an encoding function to map the scene elements in each set to their corresponding numbers, the scene element combination S=(r,e,p,b,h) can be represented as a fixed-length discrete encoding sequence, as shown in equation (13): (13) In the formula, enc(S) is a sequence composed of discrete codes; The meaning of the encoding at each position in the encoded sequence is determined by the dimension in which it is located; after discrete encoding mapping, the original set of scene element combinations S orig Converted into a set of encoded sequences S enc ; 32) Construct the HyCon-Sg-Net algorithm for deriving scene element combination sequences; The HyCon-Sg-Net algorithm is constructed to extend the combination of accident scene elements, integrating the advantages of diffusion models and generative adversarial networks, and combining multiple guidance mechanisms; the HyCon-Sg-Net algorithm uses an encoder to process the input scene element encoding S enc Then, through a forward diffusion process, a conditional generation-reverse denoising process, and with the help of multiple guidance modules, generated sample pairs are obtained. These samples are then input into the discriminator along with real sample pairs for discrimination. In conjunction with the training objective of the algorithm, a high-confidence expansion of the accident text knowledge base is achieved. In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and the log-variance vector logσ of the latent variables. 2 Furthermore, the reparameterization technique is employed to sample the latent vector x in the Gaussian distribution. T As shown in equation (14): (14) In the forward diffusion process, the diffusion probability model is used to determine the sample x0=x T x is generated by gradually adding noise within time step T. t Let β t The noise injection process at each time step is shown in Equation (15), which represents the variance increment: (15) In the formula, The obtained x t That is, the input vector for the reverse denoising process, q(x) t |x t-1 ) represents the transition probability distribution during the forward diffusion process; In the conditional generation-inverse denoising process, the goal of inverse denoising is to transform the input x under condition c. t Gradually restored to x0, the generator prediction noise is Where z is a random latent variable, and x is restored t-1 The mean is expressed as shown in equation (16): (16) In the formula, μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in Equation (17): (17) In the formula, This represents the probability distribution of the backsampling process; The variance is the learnable variance; N is a Gaussian distribution. Discriminator D φ Using real samples With generated sample pairs As input, a non-saturated adversarial target is adopted, as shown in equations (18) and (19): (18) (19) In the formula, η is the noise sampled from the Gaussian distribution, i.e. ; q represents the data distribution of the real sample; G θ The data distribution for generating samples for the generator; In the multi-guidance module, with probability P drop The random null condition c is used for joint prediction during inference based on equation (20) to coordinate the controllability and diversity of target generation: (20) In the formula, This is the result generated under unconditional c; This represents the generated result under condition c; λ1 is the weight adjustment factor, with a value between 0 and 1. In the multi-guidance module, unreasonable combinations of elements are excluded by injecting prior violation conditions; the rule evaluation function r(x0,c) is defined. g ), used to determine whether sample x0 meets the violation condition c. g Thus, the constrained guided loss is obtained as shown in equation (21): (21) In a multi-boot module, for the same x T Repeat sampling M times, and evaluate the output set using equation (22). The variance, if σ 2 >δ, where δ is the set threshold, indicates that the fluctuation between samples is too large, the model refuses to output and resamples; (22) In the formula, d is the dimension of the generated sample; This represents the value of the i-th dimension in the m-th sampling result; The overall training loss of the HyCon-Sg-Net algorithm is shown in Equation (23): (23) In the formula, L total Total loss; L diff L represents the noise prediction error of the diffusion probability model DDPM; G L D To combat the losses, L guide To constrain the guided loss, λ G , λ D , λ guide is the weighting coefficient for the loss.

5. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that, The specific method for step four is as follows: 41) Construct an Open-Sora model fine-tuning dataset, refine the text prompts in the fine-tuning dataset, and fine-tune the Open-Sora model; We collected real traffic accident videos from open-source datasets and online platforms; we manually wrote rough text descriptions for each accident video, which accurately summarized the key scene elements and the sequence of events in the video. Building upon the labeled prompts, the Video-LLaMA 2 model, a large-scale model for multimodal data understanding, is introduced to refine and enhance the coarsely labeled prompts. Accident videos and manually labeled coarse prompts are input into the model, which leverages its powerful visual semantic understanding capabilities to generate more detailed and accurate text descriptions. This supplements missing environmental details and traffic participant actions, and corrects descriptions that do not match the video footage, resulting in a finely adjusted and enhanced prompt T. ref1 ; The enhanced text description and the original accident video constitute a high-quality training pair (T). ref1 Based on this dataset, the Open-Sora model parameters θ are fine-tuned and trained. The model is updated by maximizing the log-likelihood of the real video under the text condition, as shown in Equation (24): (24) In the formula, D ft For fine-tuning the video-text dataset; p θ (V|T) represents the probability distribution of the model generating video V from text T; 42) Refinement of rule combination prompts and generation of accident video guidance; Using a knowledge graph-based accident scenario text knowledge base, accident scenario entries are queried in the text knowledge base according to the expected generation target. Then, an initial text prompt word T is generated using a rule-based accident information combination method. init The initial prompt is simply composed of various entity nodes under a root node in the knowledge graph. Then, the ChatGPT-4o large language model is used to process the initial prompt word T. init Expanded to a more detailed and refined intensifying cue word T enhance , and thus T enhance By inputting the finely tuned Open-Sora model, accident videos that match the refined prompt word descriptions are generated. This completes the transformation from an accident text knowledge base to an accident video database, resulting in a multimodal accident scenario database that can be used for targeted testing of end-to-end autonomous vehicles.

Images