An automatic driving video question answering method, model training method and system

Abstract

The application provides an automatic driving video question and answer method, a model training method and a system. A training sample set is used to train an initial automatic driving question and answer model. Each sample contains continuous video image data, audio data and question text data for an automatic driving video. Video features and audio features are fused and input into a scene hypergraph generation module. Relationship features are obtained by inputting a relationship query into a relationship decoder. Driving behavior features are obtained by inputting a driving behavior query into a driving behavior decoder. Two types of classification heads output two types of classification results. A scene hypergraph embedding layer adds corresponding classification results and time step embeddings to the relationship features and the driving behavior features. After introducing identifier embeddings, scene hypergraph embeddings are obtained. The scene hypergraph embeddings and question text features are jointly input into a video question and answer module to output answer prediction values for the question text data. An automatic driving video question and answer model is obtained by constructing a fusion loss function through a Hungarian matching loss function and a cross-entropy loss function.

Description

Technical Field

[0001] The present invention relates to the field of audio and video information processing, and in particular to an autonomous driving video question-answering method, a model training method, and a system. Background Art

[0002] With the rapid development of multimodal tasks such as video question answering and video analysis, video question answering tasks in autonomous driving scenarios enable autonomous driving video question answering models to understand the scene and select appropriate answers by analyzing video clips and combining them with user questions. This allows the model to gradually deepen its understanding of the scene through four layers: perception, prediction, planning, and behavior. However, autonomous driving video question answering models face the challenges of jointly representing cross-modal data, capturing time series information, and accurately understanding video semantics. These challenges require powerful spatiotemporal modeling capabilities and multimodal fusion techniques. In particular, in complex autonomous driving scenarios, the key research question is how to accurately model objects, driving behaviors, and scenes in the video and effectively align them with the text questions.

[0003] Existing video question-answering methods mostly employ a combination of convolutional neural networks and self-attention mechanisms, or methods based on explicit scene graph modeling, to capture relationships between objects and actions in videos, thereby better understanding complex video scenes. Alternatively, pre-trained language models are used to process text questions and answers, and multimodal alignment of image and text features is leveraged to improve the accuracy of video question-answering. Attention-based methods leverage the interaction between vision and text to answer questions. However, because these models primarily rely on static attention weights, they struggle to capture temporal information in dynamic scenes, particularly when complex temporal reasoning is involved. Research on cross-modal interactions aims to establish close connections between video and other modalities, such as text and audio. However, multimodal data alignment is difficult, and this can limit the model's reasoning capabilities when there is a significant semantic gap between video frames and language expressions or question forms. Hierarchical learning is another strategy for video question-answering, which answers questions by learning information at different levels of abstraction layer by layer. However, the drawback is that having too many layers can lead to information loss or overfitting, increasing the time cost of model training and reducing efficiency in practical applications. Modular networks and symbolic reasoning decompose the problem into multiple subtasks, each with an independent module. However, there is a lack of sufficient interaction between modules, resulting in limited performance in dealing with multimodal long-term dependency reasoning problems and poor adaptability to open-domain free question-answering scenarios. Summary of the Invention

[0004] In view of this, the embodiments of the present invention provide an autonomous driving video question-answering method, a model training method, and a system to eliminate or improve one or more defects in the prior art, and solve the problem of low answer accuracy in autonomous driving video question-answering tasks in the prior art.

[0005] One aspect of the present invention provides a method for training an autonomous driving video question-answering model, the method comprising the following steps:

[0006] Obtaining a training sample set, the training sample set comprising a plurality of samples, each sample comprising continuous video image data and audio data generated for an autonomous driving video, and textual data of questions posed regarding a scene in the autonomous driving video, with true values ​​of driving state relationships between objects in the autonomous driving video, true values ​​of driving behavior categories, and true values ​​of answers to the textual data of the questions serving as labels;

[0007] The training sample set is used to train an initial autonomous driving question-answering model, and the initial autonomous driving question-answering model includes: a video feature extraction module for extracting video features from the video image data, an audio feature extraction module for extracting audio features from the audio data, a question text feature extraction module for extracting question text features from the question text data, a visual and audio feature fusion module, a scene hypergraph generation module and a video question-answering module; the video features and the audio features are compressed and fused by the visual and audio feature fusion module to obtain audio and video fusion features; the scene hypergraph generation module includes a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head and a scene hypergraph embedding layer; the relationship embedding layer obtains a relationship query from the audio and video fusion feature, and the relationship decoder classifies the relationship query based on the cross-attention mechanism. The driving behavior embedding layer obtains a driving behavior query from the audio and video fusion features, and the driving behavior decoder obtains driving behavior features from the driving behavior query and the audio and video fusion features based on a cross-attention mechanism, and the driving behavior classification head outputs a driving behavior classification result. The scene hypergraph embedding layer adds an embedding of the corresponding relationship classification result and time step to the relationship feature, and adds an embedding of the corresponding driving behavior classification result and time step to the driving behavior feature, and splices them to obtain a scene hypergraph embedding. The scene hypergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video. The question text feature is compressed and input into the video question-answering module together with the scene hypergraph embedding to output a predicted answer value for the question text data.

[0008] The deviation of the driving state relationship true value and the relationship classification result, the deviation of the driving behavior class true value and the driving behavior classification result are used to construct a Hungarian matching loss function, a cross-entropy loss function is constructed according to the deviation of the answer true value and the answer prediction value, a fusion loss function is obtained through the Hungarian matching loss function and the cross-entropy loss function, and the initial automatic driving video question and answer model is iteratively updated in parameters to obtain an automatic driving video question and answer model.

[0009] In some embodiments, the step of obtaining the driving state relationship true value between each object in the automatic driving video comprises:

[0010] The preprocessed continuous video image data is input into a pre-trained target detection network to obtain the bounding box information and object category of each object in each video image;

[0011] The pre-trained scene graph generation network generates a scene graph of the continuous video data and obtains the object category in each video image and the driving state relationship true value between each object through the bounding box information and the object category of each object.

[0012] In some embodiments, the video feature extraction module comprises a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module comprises a spectrum graph conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module comprises an embedding layer, two layers of recurrent neural networks, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module comprises a three-layer self-attention layer and two fully connected layers; and the video question and answer module comprises a multi-layer cross-attention layer, a linear layer and a softmax function layer.

[0013] In some embodiments, when the video feature and the audio feature are compressed, the three-layer self-attention layer of the audio-visual feature fusion module is used, comprising:

[0014] The first self-attention layer obtains a first query matrix, a first key matrix and a first value matrix through three different linear transformations on the input feature, and obtains a first output feature according to a self-attention calculation formula;

[0015] The second self-attention layer obtains a second query matrix, a second key matrix and a second value matrix through three different linear transformations on the first output feature, and obtains a second output feature according to a self-attention calculation formula;

[0016] The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix through three different linear transformations on the second output feature, and obtains a third output feature according to a self-attention calculation formula.

[0017] In some embodiments, the expression of the relation decoder based on the cross attention mechanism is:

[0018] Q=R q W q , K=FW k , V=FW v ;

[0019]

[0020] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents the relational query, K represents the key matrix, W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function;

[0021] The driving behavior decoder is expressed as follows based on the cross-attention mechanism:

[0022] Q=A q W q , K=FW k , V=FW v ;

[0023]

[0024] Among them, A q represents the driving behavior query.

[0025] In some embodiments, the question text features are compressed and then input into the video question-answering module together with the scene hypergraph embedding to output the answer prediction value for the question text data, which includes:

[0026] The multi-layer cross attention layer in the video question answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs the result into the linear layer to obtain the answer classification embedding;

[0027] The softmax function layer obtains the probability of each answer to the question text data through the answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

[0028] In some embodiments, a Hungarian matching loss function is constructed based on the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result, and the expression is:

[0029] L1=λ1L rel +λ2L act ;

[0030]

[0031]

[0032] Among them, L rel represents the relationship loss formed by the deviation between the true value of the driving state relationship and the relationship classification result, λ1 represents the hyperparameter of the relationship loss, r represents the true value of the driving state relationship, Represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss formed by the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

[0033] In some embodiments, a cross entropy loss function is constructed based on the deviation between the true value of the answer and the predicted value of the answer, and the expression is:

[0034]

[0035] Wherein, yi represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

[0036] Another invention, the present invention also provides an autonomous driving video question-answering method, the method comprising:

[0037] Obtaining an autonomous driving video to be tested and a text of a question raised with respect to the autonomous driving video;

[0038] The autonomous driving video to be detected and the question text are input into the autonomous driving video question-answering model obtained by any of the above-mentioned autonomous driving video question-answering model training methods, and the answer to the question in the autonomous driving video is output.

[0039] On the other hand, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the system implements the steps of any one of the above methods.

[0040] The beneficial effects of the present invention are at least:

[0041] In the autonomous driving video question-answering method, model training method and system described in the present invention, the initial training sample set is trained by multiple samples including continuous video image data, audio data and question text data raised for scenes in the autonomous driving video generated for an autonomous driving video. The multi-layer convolutional network of the video feature extraction module and the audio feature extraction module effectively extracts important information from the video and audio. The two-layer recurrent network of the question text feature extraction module captures the contextual information of the text. The three features are dimensionally compressed through a linear layer to ensure that the feature dimensions remain consistent. The video features and the audio features are feature compressed and then fused through three layers of attention layers. The video is understood by using multiple modalities so that the autonomous driving video question-answering model obtains more information. Answer questions; the relationship classification results output by the relationship classification head and the driving behavior classification results output by the driving behavior classification head are used as prediction values ​​and label values ​​to jointly calculate the Hungarian matching loss function to improve the generation ability of the scene hypergraph generation module. The scene hypergraph generation module can capture more semantic information and context when multiple objects work together. The spliced ​​scene hypergraph embedding represents the complex relationship between multiple objects through hyperedges, so that the autonomous driving video question answering model has greater flexibility and expressiveness when processing complex scenes; the question text features are compressed and input into the video question answering module together with the scene hypergraph embedding, and the output answer prediction value and the true answer value are jointly calculated to calculate the cross entropy loss function to improve the accuracy of the video question answering module in answering questions.

[0042] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0043] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0045] Figure 1 This is a flowchart of the method for training an autonomous driving video question-answering model according to one embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of the autonomous driving video question-answering model according to one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0049] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0050] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] In existing technologies, most video question-answering methods use a combination of convolutional neural networks and self-attention mechanisms, or methods based on explicit scene graph modeling to capture the relationships between objects and actions in the video. Pre-trained language models are also used to process text questions and answers, leveraging multimodal alignment of image and text features. Attention-based methods struggle to capture temporal information in dynamic scenes and cannot accurately answer video questions. Multimodal data alignment is difficult when studying cross-modal interactions, limiting the model's reasoning capabilities. Excessive hierarchical learning layers can lead to information loss or overfitting, increase model training time, and reduce practical application efficiency. Modular networks and symbolic reasoning have limited performance in dealing with problems involving multimodal long-term dependency reasoning, and have poor adaptability to free question-answering scenarios in open domains. The present invention proposes an autonomous driving video question-answering method, a model training method, and a system. The training sample set trains the initial autonomous driving question-answering model, and each sample contains continuous video image data, audio data, and text data of questions raised about the scene in the autonomous driving video generated for an autonomous driving video. The video features and audio features extracted by the extraction module are fused after feature compression to obtain audio-video fusion features, which are input into the scene hypergraph generation module. The relationship embedding layer obtains the relationship query input, the relationship decoder obtains the relationship features, and the relationship classification head outputs the relationship classification results. The driving behavior embedding layer obtains the driving behavior query input, and the driving behavior decoder obtains the driving behavior features. , the driving behavior classification head outputs the driving behavior classification result; the scene hypergraph embedding layer adds the corresponding relationship classification result and time step embedding to the relationship feature and driving behavior feature respectively, splices and introduces an identity embedding to obtain the scene hypergraph embedding; the question text feature is compressed and input into the video question answering module together with the scene hypergraph embedding to output the answer prediction value for the question text data; the Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The cross entropy loss function is constructed according to the deviation between the true value of the answer and the predicted answer value. The fusion loss function is constructed through the two loss functions. The parameters of the initial autonomous driving video question answering model are iteratively updated with the goal of minimizing the fusion loss function to obtain the autonomous driving video question answering model.

[0053] Figure 1 This is a flowchart of the method for training an autonomous driving video question-answering model according to one embodiment of the present invention. Figure 2 This is a structural diagram of an autonomous driving video question-answering model according to an embodiment of the present invention. Specifically, this application provides a method for training an autonomous driving video question-answering model, which includes the following steps S101 to S103:

[0054] Step S101: Obtain a training sample set, the training sample set containing a plurality of samples, each sample containing continuous video image data, audio data generated for an autonomous driving video, and question text data proposed for a scene in the autonomous driving video, taking the driving state relationship true value between objects in the autonomous driving video, the driving behavior category true value, and the answer true value of the question text data as labels.

[0055] Step S102: Train the initial autonomous driving question and answer model using the training sample set, the initial autonomous driving question and answer model including a video feature extraction module for extracting video features from video image data, an audio feature extraction module for extracting audio features from audio data, a question text feature extraction module for extracting question text features from question text data, a video and audio feature fusion module, a scene supergraph generation module, and a video question and answer module; the video features and the audio features are fused by the video and audio feature fusion module to obtain audio-video fusion features after feature compression; the scene supergraph generation module includes a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head, and a scene supergraph embedding layer; the relationship embedding layer obtains a relationship query from the audio-video fusion features, the relationship decoder obtains relationship features based on a cross-attention mechanism from the relationship query and the audio-video fusion features, and the relationship classification head outputs a relationship classification result; the driving behavior embedding layer obtains a driving behavior query from the audio-video fusion features, the driving behavior decoder obtains driving behavior features based on a cross-attention mechanism from the driving behavior query and the audio-video fusion features, and the driving behavior classification head outputs a driving behavior classification result; the scene supergraph embedding layer adds embedding of the corresponding relationship classification result and time step to the relationship features, adds embedding of the corresponding driving behavior classification result and time step to the driving behavior features, and splices to obtain scene supergraph embedding; the scene supergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video; the question text features are input into the video question and answer module together with the scene supergraph embedding after feature compression to output an answer prediction value for the question text data.

[0056] Step S103: Construct a Hungarian matching loss function according to the deviation of the driving state relationship true value and the relationship classification result, and the deviation of the driving behavior category true value and the driving behavior classification result, construct a cross-entropy loss function according to the deviation of the answer true value and the answer prediction value, obtain a fusion loss function through the Hungarian matching loss function and the cross-entropy loss function, and update the parameters of the initial autonomous driving video question and answer model to obtain an autonomous driving video question and answer model.

[0057] In step S101, the continuous video image data of each sample in the training set is a set number of continuous pictures with the same interval obtained after resizing the pictures in the same autonomous driving video. The audio data in each sample is the audio of an autonomous driving video evenly divided into a set number of audio clips of the same length. The question text data in each sample is the question corresponding to an autonomous driving video.

[0058] Furthermore, the true value of the driving behavior category in each label and the true value of the answer to the question text data are obtained from a preset dataset, and the driving state relationship between each object is obtained by a pre-trained target detection network and scene graph generation network. In some embodiments, the step of obtaining the true value of the driving state relationship between each object in the autonomous driving video includes steps S1011 to S1012:

[0059] Step S1011: The pre-processed continuous video image data is input into a pre-trained object detection network to obtain the bounding box information and object type of each object in each video image.

[0060] Step S1012: The pre-trained scene graph generation network generates a scene graph of the continuous video image data through the bounding box information and object type of each object and obtains the object category in each video image and the true value of the driving state relationship between each object.

[0061] Specifically, the preprocessing of continuous video image data is to transform the dimension of the continuous video image data to adapt to the input of the subsequent video feature extraction module; the target detection network includes the region-based convolutional neural network (R-CNN) series and the YOLO series, which are used to detect objects in the input continuous video images, and each video image contains a vehicle representing the current autonomous driving; the scene graph generation network includes a spatial decoder and a temporal decoder, which generates a scene graph of each video image based on the spatial information and temporal information between each object in each video image. The spatial information includes the position, shape, size of each object and the relationship between each object. The temporal information includes the changes of each object on the timeline and the interaction with other objects during the change process. The driving state relationships between objects include the front-to-back, parallel, and cross-driving position relationships between vehicles or between vehicles and other objects; the relative speed relationships between vehicles or between vehicles and other objects (same direction and same speed, same direction and different speeds, and opposite directions); and the distance relationships between vehicles or between vehicles and other objects; driving behavior categories include the speed change behavior of vehicles or other objects, the steering behavior of vehicles or other objects, and the stopping behavior of vehicles or other objects; the answers to the question text data include 26 types of answers to all questions in the preset data set.

[0062] In step S102, in some embodiments, the video feature extraction module includes a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module includes a spectrogram conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module includes an embedding layer, a two-layer recurrent neural network, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module includes three layers of self-attention layers and two fully connected layers; the video question answering module includes multiple layers of cross attention layers, a linear layer and a softmax function layer.

[0063] Specifically, the multi-layer convolutional network contains five convolutional layers and two pooling layers. The first three convolutional layers are used to extract low-level features, including edges and textures, and the last two convolutional layers are used to extract high-level spatiotemporal information, including motion patterns and object relationships. During video feature extraction, features are obtained through a multi-layer convolutional network, flattened, and then dimensionally compressed using a linear layer to obtain video features. During audio feature extraction, each audio segment in the audio data is represented as a spectrogram, and features are obtained through a multi-layer convolutional network, and then dimensionally compressed using a linear layer to obtain audio features. During question text feature extraction, question text data is segmented and embedded, and features are obtained through a two-layer recurrent neural network. The time steps of the features are pooled to compress the length, and then flattened and dimensionally compressed using a linear layer to obtain question text features. The question text features are compressed using three self-attention layers for input into the video question-answering module.

[0064] In some embodiments, when the video features and audio features are subjected to feature compression, the three self-attention layers of the audio-visual feature fusion module are used, including steps S1021 to S1023:

[0065] Step S1021: The first self-attention layer obtains a first query matrix, a first key matrix and a first value matrix through three different linear transformations on the input features, and obtains the first output feature according to the self-attention calculation formula.

[0066] Step S1022: The second self-attention layer obtains a second query matrix, a second key matrix and a second value matrix through three different linear transformations on the first output feature, and obtains the second output feature according to the self-attention calculation formula.

[0067] Step S1023: The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix through three different linear transformations on the second output feature, and obtains the third output feature according to the self-attention calculation formula.

[0068] Specifically, the video features and audio features are input into the audio-video feature fusion module, compressed by three self-attention layers, and then fused by two fully connected layers in the audio-visual feature fusion module to obtain the audio-video fusion features. Each fully connected layer contains two linear layers and one Relu activation layer.

[0069] Furthermore, the relationship embedding generation layer includes a relationship embedding layer and a driving behavior embedding layer. The audio and video fusion features are input to the scene hypergraph generation module, which obtains relationship queries through the relationship embedding layer and driving behavior queries through the driving behavior embedding layer. The relationship decoder obtains relationship features based on the relationship query and the audio and video fusion features, and the relationship classification head outputs relationship classification results. The driving behavior decoder obtains driving behavior features based on the driving behavior query and the audio and video fusion features, and the driving behavior classification head outputs driving behavior classification results. The relationship decoder and driving behavior decoder contain multiple layers of cross-attention layers, and the relationship classification head and driving behavior classification head contain two fully connected layers. The scene hypergraph embedding layer adds embeddings corresponding to the relationship classification results and time steps, embeddings corresponding to the driving behavior classification results and time steps, and an identifier embedding representing the overall content of the autonomous driving video. This is used by the scene hypergraph generation module to learn the relationships between features across different categories and time steps. The embeddings in the scene hypergraph embedding layer are provided by the PyTorch framework, which contains a set number of learnable embeddings. The identifier embedding is used to represent the overall content of the autonomous driving video, including but not limited to the driving environment, surrounding weather, road traffic conditions, and vehicle status in the current autonomous driving video.

[0070] In some embodiments, the expression of the relation decoder based on the cross-attention mechanism is:

[0071] Q=R q W q , K=FW k , V=FW v ;

[0072]

[0073] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents a relational query, K represents the key matrix, and W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function.

[0074] The driving behavior decoder is expressed as follows based on the cross-attention mechanism:

[0075] Q=Aq W q , K=FW k , V=FW v ;

[0076]

[0077] Among them, A q Indicates a driving behavior query.

[0078] Furthermore, the video question answering module includes multiple cross-attention layers, a linear layer, and a softmax function layer, wherein the linear layer includes two fully connected layers. In some embodiments, the question text features are compressed and embedded with the scene hypergraph before being input into the video question answering module to output the answer prediction value for the question text data, including steps S1 to S2:

[0079] Step S1: The multi-layer cross-attention layer in the video question answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs them into the linear layer to obtain the answer classification embedding.

[0080] Step S2: The softmax function layer obtains the probability of each answer to the question text data through answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

[0081] Specifically, the multi-layer cross attention layer performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs them into the linear layer to obtain the answer classification embedding A:

[0082] Q=TW q , K=HW k , V=HW v ;

[0083]

[0084] A=Linear(Attention(Q,K,V));

[0085] Among them, T represents the question text features and H represents the scene hypergraph embedding.

[0086] In step S103, the Hungarian matching loss function is used to train the scene hypergraph generation module to improve the accuracy of the generated scene hypergraph. The Hungarian matching loss function includes relationship loss and driving behavior loss. The cross entropy loss function is used to train the video question answering module to improve the accuracy of the video question answering task.

[0087] In some embodiments, a Hungarian matching loss function is constructed based on the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The expression is:

[0088] L1=λ1L rel +λ2L act ;

[0089]

[0090]

[0091] Among them, L rel represents the relational loss formed by the deviation between the true value of the driving state relationship and the relationship classification result, λ1 represents the hyperparameter of the relational loss, r represents the true value of the driving state relationship, Represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss composed of the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

[0092] In some embodiments, a cross entropy loss function is constructed based on the deviation between the true answer value and the predicted answer value, and the expression is:

[0093]

[0094] Among them, yi represents the true value of the answer to the i-th question text data, represents the answer prediction value of the i-th question text data, and B represents the total number of questions.

[0095] Another invention, the present invention also provides an autonomous driving video question-answering method, the method comprising steps S201 to S202:

[0096] Step S201: Obtain the autonomous driving video to be detected and the question text posed for the autonomous driving video.

[0097] Step S202: Input the autonomous driving video to be detected and the question text into the autonomous driving video question-answering model obtained by any of the above autonomous driving video question-answering model training methods, and output the answer to the question in the autonomous driving video.

[0098] On the other hand, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the system implements the steps of any one of the above methods.

[0099] The present invention will be described below in conjunction with a specific embodiment:

[0100] 1. Obtain training sample sets and labels.

[0101] Resize each video image in each autonomous driving video to obtain F consecutive pictures P = {P1, P2, ..., P F}, each video image is preprocessed and converted into a picture p = {p1,p2,…,p F}, similarly, the audio of each autonomous driving video is evenly divided into F audio segments of the same length and no less than 1.2 seconds as the input of the feature extraction module.

[0102] First, each video image is fed into a pretrained object detection network to obtain bounding boxes (1×5) and object category labels (1×N), where N is the number of object categories. Each video image contains a vehicle at the center of the image, representing the currently autonomous vehicle. These objects are then fed into a pretrained scene graph generation network, which primarily consists of a spatial encoder and a temporal decoder, to generate a scene graph from each video image. The object categories (M×N) and driving state relationships between objects are obtained for each video image. The driving state relationship labels are then combined with the driving behavior category labels from the dataset to obtain a global label (C×R), where M is the number of objects in each video image, C is the number of object pairs in the current video image, and R is the sum of all driving behaviors and driving state relationship categories. The corresponding question and correct answer are also obtained for each autonomous driving video and used to calculate the loss.

[0103] 2. Build an autonomous driving video question-answering system.

[0104] 2.1 Construct the video feature extraction module, audio feature extraction module, question text feature extraction module, and audio-visual feature fusion module. The video feature extraction module is a multi-layer convolutional network consisting of five convolutional layers and two pooling layers. The first three layers are responsible for extracting low-level features, including but not limited to edges and textures, while the last two layers use deeper convolutional structures to capture high-level spatiotemporal information, including but not limited to motion patterns and object relationships. The audio feature extraction module first converts each audio segment into a spectrogram to capture frequency domain features, and then uses a multi-layer convolutional network consisting of five convolutional layers and two pooling layers. The feature extraction module effectively extracts important information from the video and audio. The question text feature extraction module first performs word segmentation and embedding processing, converting each question text segment into a word embedding representation. It then extracts features through a two-layer recurrent neural network structure to capture the contextual information in the question text. The output question text features are pooled to reduce the time step. The output video features, audio features, and question text features are dimensionality compressed through a linear layer to ensure that the feature dimensions remain consistent for subsequent fusion and processing. Each modality feature uses a unimodal encoder, which consists of three layers of self-attention layers, residual networks, and linear layers.

[0105] The audio-visual feature fusion module consists of three self-attention layers and two fully connected layers, each of which contains two linear layers and a ReLU activation layer.

[0106] 2.2 Constructing the Scenario Hypergraph Generation Module. The Scenario Hypergraph Generation Module consists of two relation generation networks. Each relation generation network includes a relation query embedding generation layer, a decoder, and a classification head. The relation decoder consists of multiple layers of cross-attention layers, and the relation classification head consists of two fully connected layers. The relation query embedding generation layer includes a relation embedding layer and a driving behavior embedding layer. The relation decoder includes a relation decoder and a driving behavior decoder. The relation classification head includes a relation classification head and a driving behavior classification head.

[0107] 2.3 Constructing the Video Question Answering Module. The video question answering module consists of a cross-attention module, which contains multiple cross-attention layers, followed by two fully connected layer modules and a softmax function layer.

[0108] 3. The video feature extraction module extracts video features, the audio feature extraction module extracts audio features, and the question text feature extraction module extracts question text features. For video data, the input is first video image data (16x3x224x224), and the video image data is processed by a multi-layer convolutional network to obtain video features (16x7x7x16). Since the subsequent attention network needs to process sequence data, the video features are first flattened to obtain features (16x49x16), and the feature dimension is compressed to 16x1024 through a linear layer. For audio features, each piece of audio is first represented as a spectrogram, which is processed by a multi-layer convolutional network to obtain features (16x256x128), and finally the dimension is compressed to obtain audio features (16x1024) through a linear layer; for text features, first, each piece of question text is segmented and embedded to obtain an initial word embedding representation (128x300), and then the question text feature extraction network is used to process the question text features to obtain question text features (128x512). In order to unify the processing method of video and audio features, the time steps of the question text features are pooled to compress the length to 64. Then the question text features are flattened and the dimension is compressed through a linear layer to obtain text features (16x512).

[0109] Next, the three modal features are processed through three self-attention layers respectively, which helps the audio-video fusion feature (16x1024) to further learn the temporal context relationship. In order to obtain the fusion feature of video and audio, the video feature and the audio feature are spliced and then processed through a linear layer to obtain the audio-video fusion feature F (16x1024).

[0110] 4. The audio-video fusion feature is input into the scene supergraph generation module to output the scene supergraph embedding corresponding to the autonomous driving video. First, the relationship query R q (128x512) and the driving behavior query A q (48x512) are obtained through the relationship embedding layer and the driving behavior embedding layer, and the relationship query and the driving behavior query are input into the respective relationship decoder and driving behavior decoder together with the audio-video fusion feature of the previous step, and the relationship feature R f (129x512) and the driving behavior feature A f (48x512) representing each frame of the scene are obtained through multiple cross-attention layers. Taking the relationship query as an example, the calculation of cross-attention is as follows:

[0111] Q=R q W q ,K=FW k ,V=FWv ;

[0112]

[0113] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents a relational query, K represents the key matrix, and W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function.

[0114] The relationship features and driving behavior features are passed through the relationship classification head and driving behavior classification head respectively to obtain the classification results of each relationship: relationship classification results (128×418) and driving behavior classification results (48×48).

[0115] Next, we add the embeddings of the corresponding relationship classification results and time steps to the relationship features, and add the embeddings of the corresponding driving behavior classification results and time steps to the driving behavior features. We learn the relationships at different time steps and concatenate them to obtain the scene hypergraph embedding representing the entire video. We then add a representation embedding [HG] (1×512) representing the scene hypergraph type to this scene hypergraph embedding to obtain the final scene hypergraph embedding H (177×512):

[0116] H=R s +A s +[HG];

[0117] 5. Input the scene hypergraph embedding and question text features into the video question answering network and output the final answer to the question. Input the scene hypergraph embedding and text features T into the video question answering network and pass through the multi-layer cross-attention network. The calculation of the cross-attention layer is as follows:

[0118] Q=TW q , K=HW k , V=HW v ;

[0119]

[0120] Among them, H is the scene hypergraph, T is the text feature, and then this output is passed through the linear layer to get the answer classification embedding A:

[0121] A=Linear(Attention(Q,K,V));

[0122] The answer is embedded in the softmax function to obtain the probability of each answer in the dataset. The one with the highest probability is the predicted answer value of the question text data.

[0123] 6. Construct a loss function. To train the scene hypergraph generation module's ability to generate scene hypergraphs, we use the Hungarian matching loss function. This loss function aims to enable the scene hypergraph generation module to correctly generate scene hypergraphs, thereby improving the accuracy of using the scene hypergraph embedding for video question answering tasks in subsequent stages. The loss function is calculated as follows:

[0124]

[0125] Among them, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, The predicted probability of the classification result. For the relationship set loss function L rel It is similar to the driving behavior set loss function, where r represents the true value of the driving state relationship. Represents the relationship classification result, and M represents the size of the relationship set:

[0126]

[0127] Next, we use a cross entropy loss function. The goal of this loss function is to train the video question-answering module to correctly answer questions, so that the video question-answering module gives a higher probability to the final classification of the correct answer. The expression of this loss function is:

[0128]

[0129] Among them, y i Represents the true value of the answer to the i-th question text data, represents the answer prediction value of the i-th question text data, and B represents the total number of questions.

[0130] The expression of the fusion loss function is:

[0131] L=λ1L act +λ2L rel +L vqa

[0132] Among them, λ1 represents the hyperparameter of the relationship loss, and λ2 represents the hyperparameter of the driving behavior loss.

[0133] 7. Input the test set into the trained model to obtain the answer to each question. Resize the video images from the autonomous driving videos in the test set to obtain video image data, obtain the corresponding audio data and the question text data for each video, and then input the processed features into the trained autonomous driving video question-answering model to obtain the answer to the autonomous driving video question.

[0134] In summary, the present invention provides an autonomous driving video question-answering method, a model training method and a system. Each sample in the training sample set contains continuous video image data, audio data and text data of questions raised about the scene in the autonomous driving video generated for an autonomous driving video; the video features and audio features extracted by the feature extraction module are fused after feature compression to obtain audio-video fusion features, and are input into the scene hypergraph generation module; the relationship embedding layer obtains the relationship query input, the relationship decoder obtains the relationship features, and the relationship classification head outputs the relationship classification results; the driving behavior embedding layer obtains the driving behavior query input, the driving behavior decoder obtains the driving behavior features, and the driving behavior classification head outputs the driving behavior classification results; the scene The scene hypergraph embedding layer adds the corresponding relationship classification results and time step embeddings to the relationship features and driving behavior features respectively, splices them and introduces an identity embedding to obtain the scene hypergraph embedding; the question text features are compressed and input together with the scene hypergraph embedding into the video question answering module to output the answer prediction value for the question text data; the Hungarian matching loss function is constructed based on the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The cross entropy loss function is constructed based on the deviation between the true value of the answer and the predicted answer value. The fusion loss function is constructed by the two loss functions to iteratively update the parameters of the initial autonomous driving video question answering model to obtain the autonomous driving video question answering model.

[0135] Corresponding to the above method, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the system implements the steps of the method described above.

[0136] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, a discrete hardware component, or a combination thereof.

[0137] The memory, as a non-transitory computer readable storage medium, can be configured to store a non-transitory software program, a non-transitory computer executable program and a module, such as a program instruction / module corresponding to the key shielding method of the vehicle-mounted display device in the embodiments of the present application. The processor can execute various function applications and data processing of the processor by running the non-transitory software program, the instruction and the module stored in the memory.

[0138] The memory can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0139] The one or more modules are stored in the memory and, when executed by the processor, perform the method described in the embodiments.

[0140] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0141] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0142] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0143] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for training an autonomous driving video question-answering model, characterized in that: The method comprises the following steps: Obtaining a training sample set, the training sample set comprising a plurality of samples, each sample comprising continuous video image data and audio data generated for an autonomous driving video, and textual data of questions posed regarding a scene in the autonomous driving video, with true values ​​of driving state relationships between objects in the autonomous driving video, true values ​​of driving behavior categories, and true values ​​of answers to the textual data of the questions serving as labels; The training sample set is used to train an initial autonomous driving question-answering model, and the initial autonomous driving question-answering model includes: a video feature extraction module for extracting video features from the video image data, an audio feature extraction module for extracting audio features from the audio data, a question text feature extraction module for extracting question text features from the question text data, a visual and audio feature fusion module, a scene hypergraph generation module and a video question-answering module; the video features and the audio features are compressed and fused by the visual and audio feature fusion module to obtain audio and video fusion features; the scene hypergraph generation module includes a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head and a scene hypergraph embedding layer; the relationship embedding layer obtains a relationship query from the audio and video fusion feature, and the relationship decoder classifies the relationship query based on the cross-attention mechanism. The driving behavior embedding layer obtains a driving behavior query from the audio and video fusion features, and the driving behavior decoder obtains driving behavior features from the driving behavior query and the audio and video fusion features based on a cross-attention mechanism, and the driving behavior classification head outputs a driving behavior classification result. The scene hypergraph embedding layer adds an embedding of the corresponding relationship classification result and time step to the relationship feature, and adds an embedding of the corresponding driving behavior classification result and time step to the driving behavior feature, and splices them to obtain a scene hypergraph embedding. The scene hypergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video. The question text feature is compressed and input into the video question-answering module together with the scene hypergraph embedding to output a predicted answer value for the question text data. A Hungarian matching loss function is constructed based on the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. A cross-entropy loss function is constructed based on the deviation between the true value of the answer and the predicted answer value. A fusion loss function is obtained by using the Hungarian matching loss function and the cross-entropy loss function. The parameters of the initial autonomous driving video question-answering model are iteratively updated with the goal of minimizing the fusion loss function to obtain an autonomous driving video question-answering model.

2. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The step of obtaining the true value of the driving state relationship between objects in the autonomous driving video includes: The pre-processed continuous video image data is input into a pre-trained object detection network to obtain the bounding box information and object type of each object in each video image; The pre-trained scene graph generation network generates a scene graph of the continuous video data through the bounding box information of each object and the object type and obtains the object category in each video image and the true value of the driving state relationship between each object.

3. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The video feature extraction module includes a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module includes a spectrogram conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module includes an embedding layer, a two-layer recurrent neural network, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module includes three layers of self-attention layers and two fully connected layers; the video question answering module includes multi-layer cross-attention layers, a linear layer and a softmax function layer.

4. The autonomous driving video question-answering model training method according to claim 2, characterized in that: The video features and the audio features are subjected to feature compression using the three self-attention layers of the audio-visual feature fusion module, including: The first self-attention layer obtains the first query matrix, the first key matrix and the first value matrix through three different linear transformations on the input features, and obtains the first output feature according to the self-attention calculation formula; The second self-attention layer obtains a second query matrix, a second key matrix, and a second value matrix by performing three different linear transformations on the first output feature, and obtains a second output feature according to the self-attention calculation formula; The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix through three different linear transformations on the second output features, and obtains the third output features according to the self-attention calculation formula.

5. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The expression of the relation decoder based on the cross attention mechanism is: Q=R q W q ,K=FW k ,V=FW v ; Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents the relational query, K represents the key matrix, W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function; The driving behavior decoder is expressed as follows based on the cross-attention mechanism: Q=A q W q ,K=FW k ,V=FW v ; Among them, A q represents the driving behavior query.

6. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The question text features are compressed and then embedded in the scene hypergraph before being input into the video question answering module to output the answer prediction value for the question text data. The multi-layer cross attention layer in the video question answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs the result into the linear layer to obtain the answer classification embedding; The softmax function layer obtains the probability of each answer to the question text data through the answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

7. The autonomous driving video question-answering model training method according to claim 1, characterized in that: A Hungarian matching loss function is constructed based on the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The expression is: L1=λ1Lrel+λ2Lact; Among them, L rel represents the relationship loss formed by the deviation between the true value of the driving state relationship and the relationship classification result, λ1 represents the hyperparameter of the relationship loss, r represents the true value of the driving state relationship, Represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss formed by the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

8. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The cross entropy loss function is constructed based on the deviation between the true value of the answer and the predicted value of the answer, and the expression is: Among them, y i represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

9. A method for answering questions in an autonomous driving video, characterized in that: The method comprises: Obtaining an autonomous driving video to be tested and a text of a question raised with respect to the autonomous driving video; The autonomous driving video to be detected and the question text are input into the autonomous driving video question-answering model obtained by the autonomous driving video question-answering model training method according to any one of claims 1 to 8, and the answer to the question in the autonomous driving video is output.

10. An autonomous driving video question-answering system, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 9.

Images