Multi-modal accident scene library construction method for end-to-end automatic driving test

Abstract

The invention belongs to the technical field of automatic driving test, and particularly relates to a multi-mode accident scene library construction method for end-to-end automatic driving test. The method specifically comprises the following steps: step 1, extracting text information based on an MCAT-BiLSTM-CRF algorithm; 2, designing the ontology architecture, and storing accident scene information by using a knowledge graph to obtain an accident text knowledge base; step 3, derivative expansion is carried out on the accident scene element combination based on a HyCon-Sg-Net algorithm; step 4, performing enhanced fine tuning on the Open-Sora model to realize modal conversion from the text knowledge base to the accident video database so as to construct a multi-modal accident scene library; the method can be used for testing the performance of an end-to-end automatic driving automobile in an extreme accident scene, and the adaptive capacity of an automatic driving algorithm to the extreme accident scene is remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving test, and particularly relates to a multi-modal accident scene library construction method for end-to-end automatic driving test. BACKGROUND

[0002] In recent years, automatic driving cars develop rapidly, and the end-to-end automatic driving paradigm has gradually become the mainstream of automatic driving technology, which can reduce the cumulative error of the traditional modular automatic driving architecture by directly mapping the perception sensor input signal to the underlying control signal. Since the end-to-end automatic driving car directly inputs sensor information, compared with the test task of the traditional modular architecture, higher requirements are put forward for the scene mode, and the traditional trajectory-based test scene library cannot realize the performance test of the end-to-end automatic driving.

[0003] The driving scene of the automatic driving car is endless and rich, and the long-tail scene with low occurrence probability but potential danger, such as extreme weather, unexpected events and unexpected behaviors of traffic participants, is the focus of the construction of the test scene library, but the efficiency is low and the cost is high in the process of daily data collection. Accident text reports are easy to obtain, widely sourced and abundant in quantity, which are an important source of long-tail accident scenes. The text-video multi-modal accident scene library based on the text report can effectively reduce the test cost and significantly speed up the test efficiency of the end-to-end automatic driving car.

[0004] Existing researches are difficult to realize efficient and accurate extraction and structured storage of accident key information based on accident text reports, and then realize high-quality text controllable video generation, and it is urgent to break through the existing system architecture, realize the targeted generation of accident video data based on the accurate query ability of the structured accident text knowledge base, and efficiently test the system to be tested. SUMMARY

[0005] To solve the above problems, the application provides a multi-modal accident scene library construction method for end-to-end automatic driving test, which obtains accident text key information by designing a text information efficient extraction algorithm, designs a knowledge graph storage architecture based on ontology, constructs a multi-constraint guided accident scene derivation algorithm, obtains a large-scale accident text knowledge base, and then designs an Open-Sora model reinforcement fine-tuning architecture, realizes high-fidelity construction of the accident video database under specific extreme conditions through directional query of text information, and can be used for testing the performance of the end-to-end automatic driving car under extreme accident scenes, and significantly improves the adaptability of the automatic driving algorithm to the extreme accident scene.

[0006] The technical scheme of the application is explained as follows in combination with the drawings:

[0007] The application provides a multi-modal accident scene library construction method for end-to-end automatic driving test, which comprises the following steps:

[0008] Step one, extract text information based on MCAT-BiLSTM-CRF algorithm;

[0009] First, pre-process the accident text description, including text case conversion, redundant symbol deletion and replacement, and pronoun replacement, then build MCAT-BiLSTM-CRF algorithm to realize the extraction of key information in accident text description;

[0010] Step two, design the ontology architecture, store the accident scene information using knowledge graph, and get the accident text knowledge base;

[0011] The accident scene is described in the form of scene element combination, then the ontology architecture is designed, the ontology information content of the accident scene is divided into key information metadata and additional information metadata, forming the ontology concept layer, and then according to the extracted key information of the accident scene, the information is stored in the data application layer according to the relevance of the entities in the ontology architecture, the ontology concept layer and the data application layer are mapped to each other, and the knowledge graph storage and management architecture is formed;

[0012] Step three, derive and expand the accident scene element combination based on HyCon-Sg-Net algorithm;

[0013] The scene element combination sequence stored in the knowledge graph is encoded and mapped to meet the input format of the derivation algorithm, and then the HyCon-Sg-Net derivation algorithm is constructed to expand the accident scene element combination, and on the basis of step two, the expanded accident scene text knowledge base is obtained;

[0014] Step four, strengthen the fine-tuning of Open-Sora model to realize the modal conversion from text knowledge base to accident video database, and thus the multi-modal accident scene library is constructed;

[0015] First, build the Open-Sora fine-tuning dataset, and refine the text prompt words of the fine-tuning dataset as the basis for fine-tuning the Open-Sora model; based on the accident scene text knowledge base constructed in step three, query the accident scene item in the text knowledge base according to the expected generation target, and use the rule-based accident information combination method to generate the initial text prompt word T init , and then use the ChatGPT-4o large language model to expand the initial prompt word T init to the reinforcement prompt word T enhance , guiding the fine-tuned Open-Sora model to generate accident videos, realizing the modal conversion from accident text knowledge base to accident video data.

[0016] Further, the specific method of step one is as follows:

[0017] 11) Preprocess the accident text description;

[0018] Accident text preprocessing is used to improve the standardization of accident text reports. The accident text preprocessing process includes text case conversion, redundant symbol deletion and replacement, and pronoun replacement.

[0019] First, the accident text is case-converted to unify the writing format of the accident text report and converts uppercase and lowercase characters to lowercase characters; then, redundant symbols in the accident text are deleted and replaced, and special symbols with clear meanings are replaced with words; finally, pronoun replacement is performed on the accident text, and pronouns in the text report are converted into actual referents;

[0020] 12) Construct the MCAT-BiLSTM-CRF algorithm for extracting key information from accident texts;

[0021] The MCAT-BiLSTM-CRF algorithm takes preprocessed accident text as input, splits the token sequence, and passes it through the input representation module. It then processes the accident information using a BERT encoder, a multi-scale semantic aggregation module, a context enhancement module, a BiLSTM network, and a structure-aware CRF decoder. By designing the algorithm's training objectives, it extracts key information from the accident text description.

[0022] First, the pre-processed accident text is segmented to obtain a token sequence T=[w1,w2,…,w n ], each token is represented by a function Convert to latent vector , embed token into x in the input representation module i , paragraph embeds i , position embedding p i Add together to get the input matrix of BERT encoder ;

[0023] The BERT encoder is composed of L stacked Transformer blocks, which extracts hierarchical features from the input matrix X and outputs multi-layer context representation information, as shown in formula (1):

[0024] (1)

[0025] Where, For the The hidden state of layer and position i; d is the hidden state dimension;

[0026] In the multi-scale semantic aggregation module, learnable cross-layer attention weights are introduced The outputs of different layers are weighted and aggregated as shown in equations (2) and (3):

[0027] (2)

[0028] (3)

[0029] wherein, is a cross-layer attention coefficient satisfying , m i is a fused multi-scale word vector; u k is a scoring scalar of the k-th layer hidden state; is a scoring result corresponding to the m-th layer;

[0030] In the context enhancement module, the global semantic information is explicitly modeled, and the global vector c is calculated using equation (4):

[0031] (4)

[0032] The global vector c is then residually fused as shown in equation (5):

[0033] (5)

[0034] wherein, is a concatenation between vectors; FFN is a two-layer feedforward network; is a global fused semantic;

[0035] After obtaining the global fused semantic , the global fused semantic is processed using Bi-LSTM, including two links in the forward and reverse directions, which are concatenated at position t to obtain the feature h t as shown in equation (6):

[0036] (6)

[0037] wherein, d h is the dimension of the unidirectional hidden vector;

[0038] The decoded label sequence y = [y1, …, y n ] is then explicitly distinguished into two categories in the structure-aware CRF decoder: scene elements E T and additional information A I , and the linear score in the CRF is defined as shown in equation (7):

[0039] (7)

[0040] wherein,​ is a linear scoring function; is a label weight vector; is a transition matrix, and then the conditional probability is expressed as equation (8):

[0041] (8)

[0042] where h 1:n is the hidden state sequence from the 1st to the nth token; is the predicted label sequence;

[0043] The MCAT-BiLSTM-CRF algorithm obtains the final predicted label of each token by solving the maximum sequence of equation (8), and then obtains the complete entity, and the model training objective is shown in equation (9):

[0044] (9)

[0045] where y* is the annotated label sequence.

[0046] Further, the specific method of step two is as follows:

[0047] 21) The accident scene elements are combined to describe;

[0048] The key information extracted from the text is decomposed into multi-dimensional scene elements to systematically represent the accident scene. The accident scene is often composed of the following scene elements: road type, environmental condition, traffic participant and driving behavior. Different categories of scene elements are regarded as different dimensions of describing the accident scene, and each dimension of scene element is combined with each other. When describing the accident scene, the accident scene severity is integrated into the combination of accident scene elements to reflect the influence of different combinations of scene elements on the severity of the accident scene.

[0049] The multi-dimensional scene elements are described by using the element tuple. R, E, P, B and H respectively represent the road type, environmental condition, traffic participant, driving behavior and scene severity, and then the accident scene S is represented in the form of five tuples as shown in equation (10):

[0050] (10)

[0051] wherein , , , , respectively represent the specific values of the road type, environmental condition, traffic participant, driving behavior and scene severity under a certain accident scene.

[0052] 22) Design the ontology architecture;

[0053] The ontology architecture corresponds to the ontology concept layer of the knowledge graph, and includes four concepts of class, entity, relationship and attribute in the ontology. The class is a set of entities with similar attributes, the entity is a basic component unit in the ontology, the relationship is used to define the connection between classes or entities, and the attribute is used to describe the concept of the class or entity.

[0054] The ontology architecture is constructed by the Stanford seven-step method, and the accident text data is divided into key information metadata and additional information metadata. The key information metadata is a combination of multi-dimensional elements of the accident scene, and the additional information metadata includes accident description, accident time, accident location and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to the entity correlation;

[0055] 23) store the accident information based on the knowledge graph;

[0056] Each entity in the key information metadata and the additional information metadata corresponding to the ontology architecture is regarded as an entity node in the knowledge graph, and the corresponding entities are connected according to the attribute relationship between the information, so as to represent a complete accident scene. For each key information metadata, a key information relationship triple is added in the knowledge graph, as shown in formula (11). For each additional information metadata, an additional information relationship triple is added, as shown in formula (12):

[0057] (11)

[0058] (12)

[0059] In the formula, ScenarioID i is the scene root node, which is used to represent the related information of the i th accident scene; hasKey is connected with the key information metadata; hasAdditional is connected with the additional information metadata; k is the key information metadata; a is the additional information metadata;

[0060] In the constructed knowledge graph, all the accident scene information is organized as a multi-level knowledge graph structure. Different categories of nodes are aggregated through the implicit relationship of "belonging to the same scene" to constitute a complete description of the accident scene. On this basis, Neo4j software is used to realize the storage of accident information based on the knowledge graph.

[0061] Further, the specific method of step three is as follows:

[0062] 31) encode and map the scene element combination sequence;

[0063] The combination of the semantic representation of the scene elements is converted into an algorithmically processable numerical coding form, and for each multi-dimensional element combination S=(r, e, p, b, h) under the scene of the accident, the scene elements of different dimensions are mapped to the corresponding numerical discrete coding;

[0064] Let L r , L e , L p , L b , L h represent the size of the set of scene elements of different dimensions respectively, and define the coding functions f R : r→1,…,L r , f E : e→1,…,L e , f P : p→1,…,L p , f B : b→1,…,L b , f H : h→1,…,L h , which map the scene elements in each set to the corresponding number using the coding function, and then the combination of the scene elements S=(r, e, p, b, h) can be represented as a fixed-length discrete coding sequence, as shown in equation (13):

[0065] (13)

[0066] In the formula, enc(S) is a sequence composed of discrete coding;

[0067] The coding meaning of each position in the coding sequence is determined by the dimension; after discrete coding mapping, the original scene element combination set S orig is converted into the coding sequence set S enc .

[0068] 32) Construct the scene element combination sequence derived algorithm HyCon-Sg-Net;

[0069] The construction of the HyCon-Sg-Net algorithm extends the accident scene element combination, combines the advantages of diffusion model and generative adversarial network, and combines the multiple guidance mechanism; the HyCon-Sg-Net algorithm uses the encoder to process the input scene element coding S enc , and then sequentially passes through the forward diffusion process, the conditional generation-reverse denoising process, and cooperates with the multiple guidance module, to obtain the generated sample pair, which is input into the discriminator together with the real sample pair for discrimination, and cooperates with the training target of the algorithm to realize the high-confidence expansion of the accident text knowledge base;

[0070] In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and logarithmic variance vector logσ of the latent variables 2 , and then use the reparameterization technique to achieve the sampling potential vector x in the Gaussian distribution T , as shown in formula (14):

[0071] (14)

[0072] In the forward diffusion process, the diffusion probability model is used to transform the sample x0=x T Gradually add noise to generate x in T time steps t , let β t is the variance increment, and the noise injection process at each time step is shown in formula (15):

[0073] (15)

[0074] Where, , and the obtained x t This is the input vector of the reverse denoising process, q(x t |x t-1 ) is the transition probability distribution in the forward diffusion process;

[0075] In the conditional generation-reverse denoising process, the goal of reverse denoising is to transform the input x under condition c into t Gradually restored to x0, the generator prediction noise is , where z is a random latent variable, and x is restored t-1 The mean expression of is shown in formula (16):

[0076] (16)

[0077] Where μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in formula (17):

[0078] (17)

[0079] Where, is the probability distribution of the reverse sampling process; is the learnable variance; N is the Gaussian distribution;

[0080] Discriminator D φ Using real samples With the generated sample As input, a non-saturated adversarial target is used, as shown in Equations (18) and (19):

[0081] (18)

[0082] (19)

[0083] wherein, η is the noise sampled in Gaussian distribution, i.e. q is the data distribution of real samples; G θ is the data distribution of samples generated by the generator;

[0084] In the multi-guidance module, the probability P drop is randomly set to nullify the condition c, and the controllability and diversity of the generated coordination target are jointly predicted according to formula (20) during reasoning:

[0085] (20)

[0086] wherein, is the generation result under the unconditional c; is the generation result under the conditional c; λ1 is a weight adjustment factor, and the value is between 0 and 1;

[0087] In the multi-guidance module, by injecting the prior violation condition, unreasonable element combinations are excluded; the rule evaluation function r(x0, c g ) is defined, which is used to judge whether the sample x0 satisfies the violation condition c g , and then the constraint guidance loss is obtained as shown in formula (21):

[0088] (21)

[0089] In the multi-guidance module, the same x T is repeatedly sampled M times, and the variance of the output set is evaluated by formula (22), if σ 2 > δ, δ is the set threshold, which means that the fluctuation between samples is too large, and the model rejects the output and resamples;

[0090] (22)

[0091] wherein, d is the dimension of the generated sample; is the value of the i-th dimension in the m-th sampling result;

[0092] The overall training loss of the HyCon-Sg-Net algorithm is shown in formula (23):

[0093] (23)

[0094] wherein, L total is the total loss; L diff is the noise prediction error of the diffusion probability model DDPM; L G , L DTo combat loss, L guide To constrain the guidance loss, λ G , λ D , λ guide is the weighting coefficient of the loss.

[0095] Further, the specific method of step four is as follows:

[0096] 41) Construct the Open-Sora model fine-tuning dataset, and refine the fine-tuning dataset text prompt words, fine-tune the Open-Sora model;

[0097] Real traffic accident videos are collected from open-source datasets and network platforms; artificial rough text description prompts are written for each accident video, and the rough description needs to accurately summarize the key scene elements and accident timing process contained in the video;

[0098] Based on the labeled prompts, the multi-modal data understanding large model Video-LLaMA 2 is introduced to refine and enhance the rough labeled prompts. The accident video and the artificially labeled rough prompts are input into the model, and the powerful visual semantic understanding ability is used to generate more detailed and accurate text descriptions, supplement the missing environmental details, traffic participant actions, and correct the parts of the description that do not match the picture, thereby obtaining the fine-tuned and enhanced prompt T ref1 ;

[0099] The enhanced text description and the original accident video constitute a high-quality training pair (T ref1 , V), based on which the Open-Sora model parameters θ are fine-tuned and trained, and the model is updated by maximizing the log-likelihood of the real video under the text condition, as shown in equation (24):

[0100] (24)

[0101] In the formula, D ft is the fine-tuned video-text dataset; p θ (V|T) is the probability distribution of the model generating video V from text T;

[0102] 42) Rule combination prompt refinement and accident video guided generation;

[0103] Use the accident scene text knowledge base constructed based on the knowledge graph, query the accident scene items in the text knowledge base according to the expected generation target, and use the rule-based accident information combination method to generate the initial text prompt T init , the initial prompt is simply spliced from each entity node under a root node in the knowledge graph, and then the ChatGPT-4o large language model is used to generate the initial prompt T initThe expansion is a more detailed and refined reinforcement prompt word T enhance , and T enhance is input into the Open-Sora model after fine-tuning, an accident video consistent with the refined prompt word description is generated, and thus the conversion from the accident text knowledge base to the accident video database is realized, and a multi-modal accident scene database capable of being used for end-to-end automatic driving car targeted testing is obtained.

[0104] The beneficial effects of the present application are:

[0105] The multi-modal accident scene library construction method for end-to-end automatic driving car testing provided by the present application can generate a multi-modal accident scene library including an accident text knowledge base and an accident video database according to easily obtained accident text report information, can guide the model to generate an extreme accident scene video consistent with the text description by performing directional query in the accident text knowledge base and generating a refined text prompt word, and can fully verify the performance of the end-to-end automatic driving car in the extreme accident scene and test the robustness of the end-to-end automatic driving perception algorithm in the extreme accident scene. BRIEF DESCRIPTION OF DRAWINGS

[0106] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced below, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0107] Figure 1 The flowchart of the present application;

[0108] Figure 2 The MCAT-BiLSTM-CRF algorithm architecture diagram;

[0109] Figure 3 The ontology concept layer architecture diagram;

[0110] Figure 4 The mapping relationship diagram of the ontology concept layer and the data application layer;

[0111] Figure 5 The HyCon-Sg-Net algorithm architecture diagram;

[0112] Figure 6 The Open-Sora model reinforcement fine-tuning architecture diagram;

[0113] Figure 7 The end-to-end automatic driving car target detection algorithm failure case diagram in the extreme accident scene. DETAILED DESCRIPTION

[0114] The application will be described in further detail below with reference to the drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not to be used to limit the scope of the application. In addition, it should be noted that, for the sake of brevity, only the portions of the drawings that are necessary for an understanding of the application will be described.

[0115] Embodiment one

[0116] Referring to Figure 1 The embodiment provides a multi-modal accident scene library construction method for end-to-end automatic driving test, including the following steps.

[0117] Step one, text information is extracted based on the MCAT-BiLSTM-CRF algorithm;

[0118] First, the accident text description is preprocessed, including three steps of text case conversion, redundant symbol deletion and replacement, and pronoun replacement, and then the MCAT-BiLSTM-CRF algorithm is constructed to realize the extraction of key information in the accident text description, as follows:

[0119] 11) The accident text description is preprocessed;

[0120] The accident text preprocessing is used to improve the standardization of the accident text report and reduce the inaccuracy of key information extraction due to differences in text writing methods. The process includes three steps of text case conversion, redundant symbol deletion and replacement, and pronoun replacement;

[0121] First, the accident text is converted to lowercase to unify the writing form of the accident text report, and the large characters are converted to lowercase characters; then, the redundant symbols in the accident text are deleted, such as " / ""*", to avoid affecting the key information extraction effect, and the special symbols with clear meaning are replaced with words, such as replacing the symbol "%" with "percentages"; finally, the pronouns in the accident text are replaced, such as "it" and "the", to ensure the completeness of the text report information and avoid algorithm recognition difficulties;

[0122] 12) The accident text key information extraction algorithm MCAT-BiLSTM-CRF is constructed;

[0123] Referring to Figure 2 The MCAT-BiLSTM-CRF algorithm is composed of an input representation module, a BERT encoder, a multi-scale semantic aggregation module, a context enhancement module, a BiLSTM network, and a structure perception CRF decoder.

[0124] The input representation module is used for text segmentation processing. It obtains the input matrix of the BERT encoder by adding multi-dimensional embedding vectors.

[0125] BERT encoder, which extracts hierarchical features from the input matrix and outputs multi-layer contextual representation information;

[0126] Multi-scale semantic aggregation module, used to perform weighted aggregation on the outputs of different layers;

[0127] The context enhancement module is used for display modeling of global semantic information, strengthening the ability to process cross-sentence dependency features, and improving the problem of fuzzy entity boundaries;

[0128] BiLSTM network is used to process global semantics and further characterize text temporal dependencies and entity boundaries;

[0129] The structure-aware CRF decoder is used to decode the predicted label sequence.

[0130] The MCAT-BiLSTM-CRF algorithm takes preprocessed accident text as input, splits the token sequence, and passes it through the input representation module. It then processes the accident information using a BERT encoder, a multi-scale semantic aggregation module, a context enhancement module, a BiLSTM network, and a structure-aware CRF decoder. By designing the algorithm's training objectives, it can extract key information from the accident text description.

[0131] First, the pre-processed accident text is segmented to obtain a token sequence T=[w1,w2,…,w n ], each token is represented by a function Convert to latent vector , embed token into x in the input representation module i , paragraph embeds i , position embedding p i Add together to get the input matrix of BERT encoder ;

[0132] The BERT encoder is composed of L stacked Transformer blocks, which extracts hierarchical features from the input matrix X and outputs multi-layer context representation information, as shown in formula (1):

[0133] (1)

[0134] Where, For the The hidden state of layer and position i; d is the hidden state dimension;

[0135] In the multi-scale semantic aggregation module, a learnable cross-layer attention weight is introduced The outputs of different layers are weighted and aggregated as shown in equations (2) and (3):

[0136] (2)

[0137] (3)

[0138] In the equation, is a cross-layer attention coefficient, satisfying , m i is the fused multi-scale word vector; u k is the scoring scalar of the k-th layer hidden state; is the scoring result corresponding to the m-th layer;

[0139] In the context enhancement module, the global semantic information is explicitly modeled, and the global vector c is calculated using equation (4):

[0140] (4)

[0141] The global vector c is then residually fused as shown in equation (5):

[0142] (5)

[0143] In the equation, is the concatenation between vectors; FFN is a two-layer feedforward network; is the global fused semantic, which effectively enhances the processing capability of cross-sentence dependency features, improves the problem of fuzzy entity boundaries, and is suitable for long accident text descriptions;

[0144] After obtaining the global fused semantic , a Bi-LSTM is used to process the global fused semantic, which includes two links: forward and reverse. The two links are concatenated at position t to obtain the feature h t as shown in equation (6):

[0145] (6)

[0146] In the equation, d h is the dimension of the one-way hidden vector;

[0147] Furthermore, in the structure-aware CRF decoder, the decoded label sequence y = [y1, …, y n ] is explicitly divided into two categories: scene elements E T and additional information A I ​, and define the linear score in CRF as shown in formula (7):

[0148] (7)

[0149] Where, is a linear scoring function; is the label weight vector; is the transfer matrix, and the conditional probability is expressed as formula (8):

[0150] (8)

[0151] Where h 1:n is the hidden state sequence from the 1st to the nth token; is the predicted label sequence;

[0152] The MCAT-BiLSTM-CRF algorithm obtains the final predicted label for each token by solving the maximized sequence of formula (8), and then obtains the complete entity. The model training objective is shown in formula (9):

[0153] (9)

[0154] Where y* is the labeled label sequence;

[0155] Step 2: Design the ontology architecture and use the knowledge graph to store accident scene information to obtain the accident text knowledge base;

[0156] The accident scene is described in the form of a combination of scene elements. Then, an ontology architecture is designed. The accident scene ontology information content is divided into two parts: key information metadata and additional information metadata, forming an ontology concept layer. Then, based on the extracted key information of the accident scene, the information is stored in the data application layer according to the association of entities in the ontology architecture. The ontology concept layer and the data application layer are mapped to each other to form a knowledge graph storage and management architecture, as follows:

[0157] 21) Combine and describe the elements of the accident scene;

[0158] The key information extracted from the text is decomposed into multi-dimensional scene elements to systematically represent the accident scene. The accident scene is usually composed of the following scene elements: road type, environmental conditions, traffic participants and driving behavior. The present invention regards different categories of scene elements as different dimensions of describing the accident scene, and the scene elements of each dimension are combined with each other. It should be noted that although the accident severity is not included in the scene element set E TIt is a subset of the accident severity, but it represents the accident hazard result caused by the combination of scene elements. When describing the accident scene, the accident scene severity is integrated into the accident scene element combination to reflect the impact of different scene element combinations on the accident scene severity.

[0159] The multidimensional scene elements are described using element tuples. R, E, P, B, and H represent road type, environmental conditions, traffic participants, driving behavior, and scene severity, respectively. The accident scene S can be expressed as a five-tuple as shown in formula (10):

[0160] (10)

[0161] Where, 、 、 、 、 They represent the specific values ​​of road type, environmental conditions, traffic participants, driving behavior and scene severity in a certain accident scenario;

[0162] 22) Design the ontology architecture;

[0163] The ontology architecture corresponds to the ontology concept layer of the knowledge graph. For the ontology concept layer architecture, see Figure 3 ,In ontology, there are four concepts: class, entity, relationship and attribute.,Class is a set of entities with similar attributes, entity is the basic component,of ontology, relationship is used to define the connection between,classes or entities, and attribute is used to describe the concept of,class or entity;

[0164] The ontology architecture in this invention is constructed under the guidance of Stanford's seven-step method, which divides the accident text data into two parts: key information metadata and additional information metadata. The key information metadata is a combination of multidimensional elements of the accident scene, and the additional information metadata includes additional information such as accident description, accident time, accident location, and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to the entity association. The mapping relationship between the ontology concept layer and the data application layer can be found in Figure 4 ;

[0165] 23) Storing accident information based on knowledge graph;

[0166] The present application regards each entity in the key information metadata and the additional information metadata corresponding to the ontology architecture as an entity node in the knowledge graph, and connects the corresponding entities according to the attribute relationship between the information, so as to represent a complete accident scene. Specifically, for each piece of key information metadata, a key information relationship triple is added in the knowledge graph, as shown in formula (11), and for each piece of additional information metadata, an additional information relationship triple is added, as shown in formula (12):

[0167] (11)

[0168] (12)

[0169] In the formula, ScenarioID 为 The scenario root node is used to represent the related information of the i th accident scene, “hasKey” is used to connect the scenario root node and the key information metadata, “hasAdditional” is used to connect the scenario root node and the additional information metadata, k is the key information metadata, and a is the additional information metadata;

[0170] In the constructed knowledge graph, all the accident scene information is organized as a multi-level knowledge graph structure, and different categories of nodes are aggregated through the implicit relationship of “belonging to the same scene” to constitute a complete description of the accident scene. On this basis, the present application realizes the storage of accident information based on the knowledge graph by using the Neo4j software, and introduces the semantic Web rule language into the ontology, which is helpful to realize the knowledge reasoning based on the ontology and mine the hidden information in the knowledge graph.

[0171] Step three, deriving and expanding the accident scene element combination based on the HyCon-Sg-Net algorithm;

[0172] The scenario element combination sequence stored in the knowledge graph is encoded and mapped to meet the input format of the derivation algorithm, and then the HyCon-Sg-Net derivation algorithm is constructed to expand the accident scene element combination. On the basis of step two, the expanded accident scene text knowledge base is obtained, which is as follows:

[0173] 31) The scenario element combination sequence is encoded and mapped;

[0174] The semantically represented scenario element combination is converted into a numerical coding form that can be processed by the algorithm. Specifically, for the multi-dimensional element combination S=(r,e,p,b,h) under each accident scene, the scenario elements in different dimensions are mapped into corresponding numerical discrete codes.

[0175] Let L r , L e , L p , Lb , L h respectively represent the size of different dimensional scene element sets, and define the encoding function f of each dimensional scene element R : r→1,…,L r , f E : e→1,…,L e , f P : p→1,…,L p , f B : b→1,…,L b , f H : h→1,…,L h , each scene element in each set is mapped to the corresponding number by using the encoding function, and then the scene element combination S=(r,e,p,b,h) can be represented as a fixed-length discrete encoding sequence, as shown in equation (13):

[0176] (13)

[0177] In the formula, enc(S) is a sequence composed of discrete encodings;

[0178] The encoding meaning of each position in the encoding sequence is determined by the dimension it belongs to, such as the first bit of the encoding sequence always represents the road type, and the second bit represents the environmental condition. The encoding sequence essentially preserves the semantic structure of the scene element, and after mapping through discrete encoding, the original scene element combination set S orig is converted to the encoding sequence set S enc ;

[0179] In addition, in order to ensure the completeness of the derived accident scene information, additional information metadata such as accident time and accident location are randomly extracted for combination in the original scene information library. They are only used to maintain the consistency of the overall structure when storing the knowledge graph, and do not need to focus on the accuracy of such information, which does not affect the generation of accident scene video data and the testing process of autonomous vehicles;

[0180] 32) Construct the scene element combination sequence derivation algorithm HyCon-Sg-Net;

[0181] Referring to Figure 5 , the present application constructs the HyCon-Sg-Net (Hybrid Constrained Scenario-generation Network) algorithm to expand the accident scene element combination, which combines the advantages of diffusion model and generative adversarial network and combines multiple guidance mechanisms, effectively ensuring the physical compliance and credibility of the accident scene generation; the HyCon-Sg-Net algorithm uses the encoder to process the input scene element encoding S enc, and then sequentially passes through a forward diffusion process, a conditional generation-reverse denoising process, and cooperates with a multiple guidance module to obtain a generated sample pair, which is input into a discriminator together with a real sample pair for discrimination. By using the above process and cooperating with the training target of the algorithm, high-confidence expansion of the accident text knowledge base can be realized.

[0182] In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and the logarithmic variance vector logσ of the latent variable 2 , and then uses the reparameterization technique to sample the latent variable x in the Gaussian distribution T , as shown in equation (14):

[0183] (14)

[0184] In the forward diffusion process, the diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM) is used to gradually generate x T from x0=x t in T time steps, where β t is the variance increment, and the noise injection process at each time step is shown in equation (15):

[0185] (15)

[0186] where , and the obtained x t is the input vector of the reverse denoising process; q(x t |x t-1 ) is the transition probability distribution in the forward diffusion process;

[0187] In the conditional generation-reverse denoising process, the goal of reverse denoising is to gradually restore x t under the condition c to x0, and the generator predicts the noise as , where z is a random latent variable, and the mean expression of x t-1 restoration is shown in equation (16):

[0188] (16)

[0189] where μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in equation (17):

[0190] (17)

[0191] where is the probability distribution of the reverse sampling process; is a learnable variance; N is a Gaussian distribution;

[0192] discriminator D φ with real sample pairs with generated sample pairs As input, to avoid gradient vanishing, a non-saturated adversarial objective is adopted, as shown in equations (18) and (19):

[0193] (18)

[0194] (19)

[0195] wherein, η is the noise sampled in a Gaussian distribution, i.e. ; q is the data distribution of real samples; G θ is the data distribution of samples generated by the generator;

[0196] In the multiple guidance module, the probability P drop is randomly set to empty condition c, and the joint prediction is made according to equation (20) during inference, coordinating the controllability and diversity of the generated target:

[0197] (20)

[0198] wherein, is the generation result without condition c; is the generation result under condition c; λ1 is a weight adjustment factor, taking a value between 0 and 1;

[0199] In the multiple guidance module in the HyCon-Sg-Net algorithm, by injecting prior violation conditions, unreasonable element combinations such as "heavy fog + strong light" and "dark night + strong sunlight" are excluded. Define the rule evaluation function r(x0, c g ) to judge whether the sample x0 satisfies the violation condition c g , and then obtain the constraint guidance loss as shown in equation (21):

[0200] (21)

[0201] In the multiple guidance module, the same x T is repeatedly sampled M times, and the variance of the output set is evaluated using equation (22). If σ 2 > δ (δ is a set threshold value), it means that the fluctuation between samples is too large, and the model rejects the output and resamples to improve the reliability of the output;

[0202] (22)

[0203] wherein, d represents the dimension of the generated sample, represents the value of the i-th dimension in the m-th sampling result;

[0204] The overall training loss of the HyCon-Sg-Net algorithm is shown in formula (23):

[0205] (23)

[0206] In the formula, L total represents the total loss, L diff represents the noise prediction error of the diffusion probability model DDPM, L G , L D is the adversarial loss, L guide is the constraint guidance loss, λ G , λ D , λ guide is the weighting coefficient of the loss.

[0207] Step four, reinforcement fine-tuning is performed on the Open-Sora model to realize the modal conversion from the text knowledge base to the accident video database, so as to construct a multi-modal accident scene library;

[0208] Referring to Figure 6 , first, an Open-Sora fine-tuning data set is constructed, and the text prompt words of the fine-tuning data set are refined as a basic fine-tuning Open-Sora model; based on the accident scene text knowledge base constructed in step three, the accident scene items are queried in the text knowledge base according to the expected generation target, and an initial text prompt word T init is generated by using a rule-based accident information combination method, and then the initial prompt word T init is expanded to a reinforcement prompt word T enhance by using a ChatGPT-4o large language model, so as to guide the fine-tuned Open-Sora model to generate an accident video, realize the modal conversion from the accident text knowledge base to the accident video data, and the specific process is as follows:

[0209] 41) Construct an Open-Sora model fine-tuning data set, refine the text prompt words of the fine-tuning data set, and fine-tune the Open-Sora model;

[0210] The present application collects real traffic accident videos from open source data sets and network platforms, and ensures the quality of the fine-tuning data set by manually screening to ensure clear picture quality, then uses artificial to write rough text description prompt words for each accident video, which roughly describes the key scene elements and accident time sequence process contained in the video;

[0211] On the basis of artificial labeling of prompt words, a large model Video-LLaMA 2 for multi-modal data understanding is introduced to refine and enhance the roughly labeled prompt words. The accident video and the roughly labeled prompt words are input into the model, and the powerful visual semantic understanding ability of the model is used to generate more detailed and accurate text descriptions, supplement the missing environmental details and the actions of traffic participants, and correct the parts of the description that do not match the picture, so as to obtain the fine-tuned and enhanced prompt words T ref1 ;

[0212] The text description after enhancement and the original accident video constitute a high-quality training pair (T ref1 , V). Based on this dataset, the Open-Sora model parameters θ are fine-tuned in this paper, and the model is updated by maximizing the log-likelihood of the real video under the text condition, as shown in equation (24):

[0213] (24)

[0214] In the equation, D ft represents the fine-tuned video-text dataset, p θ (V|T) represents the probability distribution of the model generating video V from text T. The optimization process minimizes the difference between the generated video and the real video, so that the model gradually learns to generate realistic accident videos under the accident text description prompt words.

[0215] 42) Rule combination prompt word refinement and accident video guided generation;

[0216] The accident scene text knowledge base is constructed based on the knowledge graph. According to the expected generation target, the accident scene entries are queried in the text knowledge base, and the rule-based accident information combination method is used to generate the initial text prompt T init The initial prompt is simply spliced from each entity node under a root node in the knowledge graph. Then, the ChatGPT-4o large language model is used to expand the initial prompt word T init to a more detailed and refined enhanced prompt word T enhance , and then input the T enhance into the fine-tuned Open-Sora model to generate an accident video consistent with the description of the refined prompt word. Thus, the conversion from the accident text knowledge base to the accident video database is realized, and a multi-modal accident scene database that can be used for end-to-end automatic driving car targeted testing is obtained.

[0217] Embodiment Two

[0218] The method provided in Example 1 is used to construct a multi-modal accident scene database, and the robustness of the automatic driving car perception algorithm in the extreme accident scene is tested. The accident scene is often sudden and is a typical extreme scene type. The perception algorithm carried by the host vehicle successfully identifies the potential dangerous vehicle within a certain distance, which is a necessary prerequisite for the host vehicle to avoid danger. If the perception algorithm of the host vehicle continues to fail, it indicates that the algorithm has certain performance defects, which is prone to cause accidents. The extreme accident scene in the present application refers to a scene that has a major challenge to the robust identification of the perception algorithm, and has high-challenge scene elements such as night, strong glare, and heavy fog.

[0219] The present application relies on the semantic query capability of the accident text knowledge base, and 100 accident scene videos under extreme conditions are generated as detection dataset 1. Specifically, the extreme accident scene cases are queried and extracted from the accident text knowledge base stored in the form of a knowledge graph, and prompt words such as "the host vehicle drives in the dark without light, the front black vehicle slows down, and the host vehicle collides with it" are generated to guide the fine-tuned Open-Sora model to generate accident videos. 100 accident videos are randomly selected from the accident data set used for model fine-tuning as detection data set 2 for algorithm effect comparison. The YOLOv8 and DeepSORT algorithms are used to identify and track dangerous vehicles in the accident scene, and the detection success rate of the host vehicle perception algorithm under different detection data sets is shown in Table 1.

[0220] Table 1 Perception algorithm performance under different data sets

[0221] Dataset Number of successful detections Success rate of detection Dataset 1 36 / 100 0.36 Dataset 2 84 / 100 0.84

[0222] As can be seen from Table 1, the extreme accident cases are queried in the text knowledge base, and then the accident video data under extreme conditions is generated, which poses a great challenge to the performance of the vehicle perception algorithm. Compared with the publicly available accident video data set, the detection success rate is reduced by up to 48%, indicating that the extreme accident scene video generated by the present application can reveal the functional limitations of the perception algorithm and improve the perception ability of the end-to-end automatic driving car in the extreme accident scene.

[0223] The failure case of the end-to-end automatic driving car target detection algorithm in the extreme accident scene is described in Figure 7In the case of the perception failure, the white vehicle approaches the host vehicle from the right side, and the perception algorithm fails to successfully identify the white vehicle during the high-speed approach of the white vehicle to the host vehicle due to the similarity between the white vehicle and the white background of the snowy day (frames 3, 22, and 41). At frames 3 and 22, the host vehicle still has the opportunity to take action to avoid and is within the safe operating range, but when at frame 41, the distance between the vehicles is close and is in a highly dangerous situation. The perception algorithm of the host vehicle successfully identifies the white vehicle at frames 57 and 58, but at this time, the white vehicle is very close to the host vehicle, and the host vehicle cannot avoid in time. This scenario reveals the detection defects of the perception algorithm for white vehicles in snowy conditions.

[0224] In summary, the method for constructing a multi-modal accident scene library for end-to-end autonomous vehicle testing provided by the present application can generate a multi-modal accident scene library including an accident text knowledge base and an accident video database according to easily accessible and widely sourced accident text reports. By performing a directional query in the text knowledge base and generating refined text prompt words, the model is guided to generate extreme accident scene videos consistent with the text description. The performance of the end-to-end autonomous vehicle in extreme accident scenes can be fully verified, and the robustness of the end-to-end autonomous perception algorithm in extreme accident scenes can be tested.

[0225] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements, and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing, characterized in that: The following steps are involved: Step 1: Extract text information based on the MCAT-BiLSTM-CRF algorithm; First, the accident text description is preprocessed, including three steps: text case conversion, redundant symbol removal and replacement, and pronoun replacement. Then, an MCAT-BiLSTM-CRF algorithm is constructed to extract key information from the accident text description. Step 2: Design the ontology architecture and use the knowledge graph to store accident scene information to obtain the accident text knowledge base; The accident scene is described in the form of a combination of scene elements. Then, an ontology architecture is designed to divide the accident scene ontology information content into two parts: key information metadata and additional information metadata, forming an ontology concept layer. Then, based on the extracted key information of the accident scene, the information is stored in the data application layer according to the relevance of entities in the ontology architecture. The ontology concept layer and the data application layer are mapped to each other to form a knowledge graph storage and management architecture. Step 3: Derivatively expand the combination of accident scene elements based on the HyCon-Sg-Net algorithm; The scene element combination sequence stored in the knowledge graph is encoded and mapped to meet the input format of the derivative algorithm, and then the HyCon-Sg-Net derivative algorithm is constructed to expand the accident scene element combination. Based on step 2, an expanded accident scene text knowledge base is obtained; Step 4: Strengthen and fine-tune the Open-Sora model to achieve modal conversion from the text knowledge base to the accident video database, thereby constructing a multimodal accident scene library; First, the Open-Sora fine-tuning dataset is constructed, and the text prompt words of the fine-tuning dataset are refined as the basis for fine-tuning the Open-Sora model. Based on the accident scene text knowledge base constructed in step 3, the accident scene entries are searched in the text knowledge base according to the expected generation target, and the rule-based accident information combination method is used to generate the initial text prompt words T init , and then use the ChatGPT-4o language model to convert the initial prompt word T init Expanded to strengthen the prompt word T enhance , guiding the fine-tuned Open-Sora model to generate accident videos, realizing the modal conversion from accident text knowledge base to accident video data.

2. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that: The specific method of step one is as follows: 11) Preprocess the accident text description; Accident text preprocessing is used to improve the standardization of accident text reports. The accident text preprocessing process includes text case conversion, redundant symbol deletion and replacement, and pronoun replacement; First, the accident text is case-converted to unify the writing format of the accident text report and converts uppercase and lowercase characters to lowercase characters; then, redundant symbols in the accident text are deleted and replaced, and special symbols with clear meanings are replaced with words; finally, pronoun replacement is performed on the accident text, and pronouns in the text report are converted into actual referents; 12) Construct the MCAT-BiLSTM-CRF algorithm for extracting key information from accident texts; The MCAT-BiLSTM-CRF algorithm takes preprocessed accident text as input, splits the token sequence, and passes it through the input representation module. It then processes the accident information using a BERT encoder, a multi-scale semantic aggregation module, a context enhancement module, a BiLSTM network, and a structure-aware CRF decoder. By designing the algorithm's training objectives, it extracts key information from the accident text description. First, the pre-processed accident text is segmented to obtain a token sequence T=[w1,w2,…,w n ], each token is represented by a function Convert to latent vector , embed token into x in the input representation module i , paragraph embeds i , position embedding p i Add together to get the input matrix of BERT encoder ; The BERT encoder is composed of L stacked Transformer blocks, which extracts hierarchical features from the input matrix X and outputs multi-layer context representation information, as shown in formula (1): (1) Where, For the The hidden state of layer and position i; d is the hidden state dimension; In the multi-scale semantic aggregation module, learnable cross-layer attention weights are introduced , perform weighted aggregation on the outputs of different layers, as shown in formulas (2) and (3): (2) (3) Where, is the cross-layer attention coefficient, satisfying , m i is the fused multi-scale word vector; u k is the scoring scalar of the k-th layer hidden state; For the The scoring results corresponding to the layer; In the context enhancement module, the global semantic information is explicitly modeled and the global vector c is calculated using formula (4): (4) Then, the global vector c is fused in a residual manner, as shown in formula (5): (5) Where, is the concatenation between vectors; FFN is a two-layer feedforward network; It is a global fusion semantics; In obtaining global fusion semantics Finally, Bi-LSTM is used to process the global fusion semantics, including two links, forward and reverse, which are spliced ​​at position t to obtain feature h t , as shown in formula (6): (6) Where, d h is the dimension of the one-way latent vector; Then, in the structure-aware CRF decoder, the decoded label sequence y=[y1,…,y n ]The display area is divided into two categories: scene elements E T With additional information A I , and define the linear score in CRF as shown in formula (7): (7) Where, is a linear scoring function; is the label weight vector; is the transfer matrix, and the conditional probability is expressed as formula (8): (8) Where h 1:n is the hidden state sequence from the 1st to the nth token; is the predicted label sequence; The MCAT-BiLSTM-CRF algorithm obtains the final predicted label for each token by solving the maximized sequence of formula (8), and then obtains the complete entity. The model training objective is shown in formula (9): (9) Where y* is the labeled label sequence.

3. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that: The specific method of step 2 is as follows: 21) Combine and describe the elements of the accident scene; Decompose the key information extracted from the text into multi-dimensional scene elements to systematically represent the accident scene. Accident scenes are often composed of the following scene elements: road type, environmental conditions, traffic participants, and driving behavior. Different categories of scene elements are regarded as different dimensions of describing the accident scene, and scene elements of each dimension are combined with each other. When describing the accident scene, the accident scene severity is integrated into the combination of accident scene elements to reflect the impact of different scene element combinations on the accident scene severity. The multidimensional scene elements are described using element tuples. R, E, P, B, and H are road type, environmental conditions, traffic participants, driving behavior, and scene severity, respectively. The accident scene S is then expressed as a five-tuple as shown in formula (10): (10) Where, 、 、 、 、 They are the specific values ​​of road type, environmental conditions, traffic participants, driving behavior and scene severity in a certain accident scenario; 22) Design the ontology architecture; The ontology architecture corresponds to the ontology concept layer of the knowledge graph. Ontology includes four concepts: class, entity, relationship, and attribute. A class is a set of entities with similar attributes. Entity is the basic component unit in the ontology. Relationships are used to define the connection between classes or entities. Attributes are used to describe the concept of a class or entity. The ontology module was constructed using the Stanford Seven-Step Method, which divides accident text data into two parts: key information metadata and additional information metadata. Key information metadata is a combination of multidimensional elements of the accident scene, while additional information metadata includes accident description, accident time, accident location, and accident title. After constructing the ontology concept layer, the extracted accident scene information is stored in the data application layer of the knowledge graph according to entity relevance. 23) Storing accident information based on knowledge graph; Each entity in the key information metadata and additional information metadata corresponding to the ontology architecture is regarded as an entity node in the knowledge graph, and the corresponding entities are connected according to the attribute relationship between the information to represent a complete accident scenario. For each key information metadata, a key information relationship triple is added to the knowledge graph, as shown in formula (11). For each additional information metadata, an additional information relationship triple is added, as shown in formula (12): (11) (12) Where, ScenarioID i is the scene root node, which is used to represent the relevant information of the i-th accident scene; hasKey is the connection between the scene root node and the key information metadata; hasAdditional is the connection between the scene root node and the additional information metadata; k is the key information metadata; a is the additional information metadata; In the constructed knowledge graph, all accident scenario information is organized into a multi-level knowledge graph structure. Nodes of different categories are aggregated through the implicit relationship of "belonging to the same scenario" to form a complete description of the accident scenario. On this basis, Neo4j software is used to implement knowledge graph-based accident information storage.

4. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that: The specific method of step three is as follows: 31) Encode and map the scene element combination sequence; The semantically represented scene element combination is converted into a numerical coding form that can be processed by the algorithm. For each multidimensional element combination S = (r, e, p, b, h) in each accident scene, the scene elements of different dimensions are mapped into corresponding numerical discrete codes; Let L r 、L e 、L p 、L b 、L h Represent the size of the scene element set of different dimensions, and define the encoding function f of the scene element of each dimension R :r→1,…,L r 、f E :e→1,…,L e 、f P :p→1,…,L p 、f B :b→1,…,L b 、f H :h→1,…,L h , use the encoding function to map the scene elements in each set to the corresponding numbers, and then the scene element combination S = (r, e, p, b, h) can be expressed as a discrete coding sequence of fixed length, as shown in formula (13): (13) Where enc(S) is a sequence consisting of discrete codes; The encoding meaning of each position in the encoding sequence is determined by the dimension in which it is located; after discrete encoding mapping, the original scene element combination set S orig is converted into a set of coding sequences S enc ; 32) Constructing the scene element combination sequence derivation algorithm HyCon-Sg-Net; The HyCon-Sg-Net algorithm is constructed to expand the combination of accident scene elements, integrating the advantages of diffusion model and generative adversarial network and combining multiple guidance mechanisms; the HyCon-Sg-Net algorithm uses the encoder to process the input scene element encoding S enc , and then sequentially undergoes the forward diffusion process, the conditional generation-backward denoising process and cooperates with the multi-guidance module to obtain the generated sample pairs, which are input into the discriminator together with the real sample pairs for discrimination. In conjunction with the algorithm's training objectives, a high-confidence expansion of the accident text knowledge base is achieved; In the encoder, a variational bottleneck structure is introduced, and the encoder predicts the mean vector μ and logarithmic variance vector logσ of the latent variables 2 , and then use the reparameterization technique to achieve the sampling potential vector x in the Gaussian distribution T , as shown in formula (14): (14) In the forward diffusion process, the diffusion probability model is used to transform the sample x0=x T Gradually add noise to generate x in T time steps t , let β t is the variance increment, and the noise injection process at each time step is shown in formula (15): (15) Where, , and the obtained x t That is the input vector of the reverse denoising process, q(x t |x t-1 ) is the transition probability distribution in the forward diffusion process; In the conditional generation-reverse denoising process, the goal of reverse denoising is to transform the input x under condition c into t Gradually restored to x0, the generator prediction noise is , where z is a random latent variable, and x is restored t-1 The mean expression of is shown in formula (16): (16) Where μ is the mean function in the reverse denoising process, and the reverse denoising sampling process is constructed as shown in formula (17): (17) Where, is the probability distribution of the reverse sampling process; is the learnable variance; N is the Gaussian distribution; Discriminator D φ Using real samples With the generated sample As input, a non-saturated adversarial target is used, as shown in Equations (18) and (19): (18) (19) Where, , η is the noise sampled from the Gaussian distribution, that is ; q is the data distribution of the real sample; G θ The data distribution for generating samples for the generator; In the multi-guide module, with probability P drop The condition c is randomly left blank, and the prediction is made according to formula (20) during inference to coordinate the controllability and diversity of target generation: (20) Where, is the generated result under unconditional c; is the generated result under condition c; λ1 is the weight adjustment factor, which ranges from 0 to 1; In the multi-guidance module, by injecting prior violation conditions, unreasonable element combinations are excluded; the rule evaluation function r(x0,c g ), used to determine whether the sample x0 meets the violation condition c g , and then the constraint guidance loss is obtained as shown in formula (21): (21) In the multi-boot module, for the same x T Repeat the sampling M times and use formula (22) to evaluate the output set The variance of 2 >δ, δ is the set threshold, indicating that the fluctuation between samples is too large, the model refuses to output and resamples; (22) Where d is the dimension of the generated sample; is the value of the i-th dimension in the m-th sampling result; The overall training loss of the HyCon-Sg-Net algorithm is shown in formula (23): (23) Where, L total is the total loss; L diff is the noise prediction error of the diffusion probability model DDPM; L G 、L D To combat the loss, L guide is the constraint guidance loss, λ G ,λ D ,λ guide is the weighting coefficient of the loss.

5. The method for constructing a multimodal accident scenario library for end-to-end autonomous driving testing according to claim 1, characterized in that: The specific method of step 4 is as follows: 41) Construct an Open-Sora model fine-tuning dataset, refine the text prompt words in the fine-tuning dataset, and fine-tune the Open-Sora model; Real traffic accident videos were collected from open source datasets and online platforms. A rough text description was manually written for each accident video. The rough description accurately summarized the key scene elements and the accident chronological process contained in the video. Based on the labeled prompt words, we introduced Video-LLaMA 2, a large model for multimodal data understanding, to refine and enhance the roughly labeled prompt words. We input the accident video and the roughly labeled prompt words into the model, and use the powerful visual semantic understanding ability to generate more detailed and accurate text descriptions, supplement the environmental details and traffic participant actions that are omitted in the annotations, and correct the descriptions that do not match the picture, thus obtaining the finely adjusted enhanced prompt word T. ref1 ; The enhanced text description and the original accident video constitute a high-quality training pair (T ref1 ,V), fine-tune the Open-Sora model parameters θ based on this dataset, and update the model by maximizing the log-likelihood of the real video under text conditions, as shown in Equation (24): (24) Where D ft Video-text dataset for fine-tuning; p θ (V|T) is the probability distribution of the model generating video V from text T; 42) Rule combination prompt word refinement and accident video guidance generation; Using the accident scene text knowledge base built based on the knowledge graph, we search for accident scene entries in the text knowledge base according to the expected generation target, and use the rule-based accident information combination method to generate the initial text prompt word T init The initial prompt is simply spliced ​​together by the entity nodes under a root node in the knowledge graph, and then the initial prompt word T is transformed into init Expanded to more detailed and refined reinforcement prompt words T enhance , and then T enhance The fine-tuned Open-Sora model is input to generate accident videos that match the refined prompt word descriptions. This realizes the transformation from an accident text knowledge base to an accident video database, and obtains a multimodal accident scenario database that can be used for targeted testing of end-to-end autonomous driving vehicles.

Images