Automatic driving video question and answer method, model training method and system

Abstract

The invention provides an automatic driving video question-answering method, a model training method and a model training system, a training sample set is used for training an initial automatic driving question-answering model, and each sample comprises continuous video image data, audio data and question text data for an automatic driving video; the extracted video features and audio features are fused and input into a scene hypergraph generation module, relation features are obtained through relation query input into a relation decoder, driving behavior features are obtained through driving behavior query input into a driving behavior decoder, and two classification heads output two classification results; the scene hypergraph embedding layer adds corresponding classification results and time step embedding to the relation features and the driving behavior features, scene hypergraph embedding is obtained after identification embedding is introduced, the scene hypergraph embedding and question text features are jointly input into a video question answering module, and an answer prediction value about question text data is output; and constructing a fusion loss function through the Hungary matching loss function and the cross entropy loss function to obtain an automatic driving video question and answer model.

Description

Technical Field

[0001] The present invention relates to the field of audio and video information processing, and in particular to an autonomous driving video question-answering method, a model training method and a system. Background Art

[0002] With the rapid development of multimodal tasks such as video question answering and video analysis, the video question answering task in the autonomous driving scenario analyzes video clips and combines user questions to allow the autonomous driving video question answering model to understand the scene and select reasonable answers. Through the four levels of perception, prediction, planning and behavior, the model gradually deepens its understanding of the scene. However, the autonomous driving video question answering model faces the challenges of joint representation of cross-modal data, capture of time series information, and accurate understanding of video semantics. It requires powerful spatiotemporal modeling capabilities and multimodal fusion technology. Especially in complex scenes of autonomous driving, how to accurately model objects, driving behaviors and scenes in the video and effectively align them with text questions has become the key to research.

[0003] In the prior art, most video question-answering methods use a combination of convolutional neural networks and self-attention mechanisms or methods based on explicit scene graph modeling to capture the relationship between objects and objects, objects and actions in the video, so as to better understand complex video scenes; on the other hand, a pre-trained language model is used to process text questions and answers, and the multimodal alignment of image and text features is used to improve the accuracy of video question-answering. The attention mechanism-based method uses the interaction between vision and text to answer questions, but because the model mainly relies on static attention weights, it is difficult to capture the timing information in dynamic scenes, especially when it involves complex time reasoning, it is impossible to obtain accurate answers to video questions. When studying cross-modal interaction, a close connection is established between video and modalities such as text and audio, but multimodal data alignment is difficult. When there is a large semantic gap between video frames and language expressions or question forms, the reasoning ability of the model will be limited. Hierarchical learning is another strategy for video question-answering. It answers questions by learning information at different abstract levels layer by layer. The disadvantage is that too many layers will cause information loss or overfitting, increase the time cost of model training, and reduce the efficiency of practical applications. Modular networks and symbolic reasoning decompose the problem into multiple subtasks, each of which has an independent module. However, there is a lack of sufficient interaction between the modules. Their performance is limited when dealing with multimodal long-term dependency reasoning problems, and they have poor adaptability to open-domain free question-answering scenarios. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide an autonomous driving video question and answer method, a model training method and a system to eliminate or improve one or more defects existing in the prior art, and solve the problem of low answer accuracy in autonomous driving video question and answer tasks in the prior art.

[0005] One aspect of the present invention provides a method for training an autonomous driving video question-answering model, the method comprising the following steps:

[0006] Obtain a training sample set, wherein the training sample set includes a plurality of samples, each sample includes continuous video image data, audio data, and question text data generated for a scene in the autonomous driving video, and uses the true value of the driving state relationship between objects in the autonomous driving video, the true value of the driving behavior category, and the true value of the answer to the question text data as labels;

[0007] The training sample set is used to train an initial autonomous driving question-answering model, and the initial autonomous driving question-answering model includes: a video feature extraction module for extracting video features from the video image data, an audio feature extraction module for extracting audio features from the audio data, a question text feature extraction module for extracting question text features from the question text data, a visual and audio feature fusion module, a scene hypergraph generation module, and a video question-answering module; the video features and the audio features are compressed and fused by the visual and audio feature fusion module to obtain audio-video fusion features; the scene hypergraph generation module includes a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head, and a scene hypergraph embedding layer; the relationship embedding layer obtains a relationship query from the audio-video fusion feature, and the relationship decoder performs a cross-attention on the relationship query based on the relationship query. The driving behavior embedding layer obtains a driving behavior query from the audio and video fusion feature, and the driving behavior decoder obtains a driving behavior feature from the driving behavior query and the audio and video fusion feature based on a cross attention mechanism, and the driving behavior classification head outputs a driving behavior classification result; the scene hypergraph embedding layer adds an embedding of the corresponding relationship classification result and time step to the relationship feature, and adds an embedding of the corresponding driving behavior classification result and time step to the driving behavior feature, and splices them to obtain a scene hypergraph embedding; the scene hypergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video; the question text feature is compressed and input into the video question-answering module together with the scene hypergraph embedding to output a predicted value of the answer to the question text data;

[0008] A Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. A cross entropy loss function is constructed according to the deviation between the true value of the answer and the predicted answer value. A fusion loss function is obtained by using the Hungarian matching loss function and the cross entropy loss function. The parameters of the initial autonomous driving video question and answer model are iteratively updated with the goal of minimizing the fusion loss function to obtain an autonomous driving video question and answer model.

[0009] In some embodiments, the step of obtaining the true value of the driving state relationship between objects in the autonomous driving video includes:

[0010] The pre-processed continuous video image data is input into a pre-trained object detection network to obtain the bounding box information and object type of each object in each video image;

[0011] The pre-trained scene graph generation network generates a scene graph of the continuous video data through the bounding box information of each object and the object type and obtains the object category in each video image and the true value of the driving state relationship between each object.

[0012] In some embodiments, the video feature extraction module includes a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module includes a spectrogram conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module includes an embedding layer, two layers of recurrent neural networks, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module includes three layers of self-attention layers and two fully connected layers; the video question-answering module includes multiple layers of cross-attention layers, a linear layer and a softmax function layer.

[0013] In some embodiments, the video features and the audio features are feature compressed using three self-attention layers of the audio-visual feature fusion module, including:

[0014] The first self-attention layer obtains the first query matrix, the first key matrix and the first value matrix through three different linear transformations on the input features, and obtains the first output feature according to the self-attention calculation formula;

[0015] The second self-attention layer obtains a second query matrix, a second key matrix and a second value matrix by performing three different linear transformations on the first output feature, and obtains a second output feature according to the self-attention calculation formula;

[0016] The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix for the second output feature through three different linear transformations, and obtains the third output feature according to the self-attention calculation formula.

[0017] In some embodiments, the expression of the relation decoder based on the cross attention mechanism is:

[0018] Q=R q W q , K=FW k , V=FW v ;

[0019]

[0020] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents the relational query, K represents the key matrix, W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function;

[0021] The driving behavior decoder is expressed based on the cross attention mechanism as follows:

[0022] Q=A q W q , K=FW k , V=FW v ;

[0023]

[0024] Among them, A q represents the driving behavior query.

[0025] In some embodiments, the question text features are compressed and then embedded in the scene hypergraph and input into the video question-answering module to output the answer prediction value about the question text data, including:

[0026] The multi-layer cross attention layer in the video question answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs the result into the linear layer to obtain the answer classification embedding;

[0027] The softmax function layer obtains the probability of each answer to the question text data through the answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

[0028] In some embodiments, a Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result, and the expression is:

[0029] L1=λ1L rel +λ2L act ;

[0030]

[0031]

[0032] Among them, L rel represents the relation loss formed by the deviation between the true value of the driving state relationship and the relationship classification result, λ1 represents the hyperparameter of the relation loss, r represents the true value of the driving state relationship, represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss formed by the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

[0033] In some embodiments, a cross entropy loss function is constructed based on the deviation between the true value of the answer and the predicted value of the answer, and the expression is:

[0034]

[0035] Wherein, yi represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

[0036] Another invention, the present invention also provides an autonomous driving video question-answering method, the method comprising:

[0037] Obtaining an autonomous driving video to be detected and a text of a question raised with respect to the autonomous driving video;

[0038] The autonomous driving video to be detected and the question text are input into the autonomous driving video question and answer model obtained by any of the autonomous driving video question and answer model training methods described above, and the answer to the question in the autonomous driving video is output.

[0039] On the other hand, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any one of the above-mentioned methods.

[0040] The beneficial effects of the present invention are at least:

[0041] In the autonomous driving video question-and-answer method, model training method and system described in the present invention, an initial training sample set is trained by multiple samples including continuous video image data, audio data and question text data posed for a scene in the autonomous driving video. The multi-layer convolutional network of the video feature extraction module and the audio feature extraction module effectively extracts important information from the video and audio. The two-layer recurrent network of the question text feature extraction module captures the contextual information of the text. The three features are dimensionally compressed through a linear layer to ensure that the feature dimensions remain consistent. The video features and the audio features are fused after feature compression through three attention layers, and the video is understood using multiple modalities so that the autonomous driving video question-and-answer model obtains more information. Answer questions; the relationship classification results output by the relationship classification head and the driving behavior classification results output by the driving behavior classification head are used as prediction values ​​and label values ​​to jointly calculate the Hungarian matching loss function to improve the generation ability of the scene hypergraph generation module. The scene hypergraph generation module can capture more semantic information and context when multiple objects work together. The spliced ​​scene hypergraph embedding represents the complex relationship between multiple objects through hyperedges, so that the autonomous driving video question-answering model has stronger flexibility and expressiveness when processing complex scenes; the question text features are compressed and jointly input into the video question-answering module with the scene hypergraph embedding, and the output answer prediction value and the true answer value jointly calculate the cross entropy loss function to improve the accuracy of the video question-answering module in answering questions.

[0042] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0043] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0045] Figure 1 The figure is a flowchart of a method for training an autonomous driving video question-answering model according to an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the structure of the autonomous driving video question-answering model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0049] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0050] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] In the prior art, most video question-answering methods use a combination of convolutional neural networks and self-attention mechanisms, or methods based on explicit scene graph modeling to capture the relationship between objects and objects, objects and actions in the video; on the other hand, pre-trained language models are used to process text questions and answers, and multimodal alignment of image and text features is used. Methods based on attention mechanisms are difficult to capture temporal information in dynamic scenes and cannot obtain accurate answers to video questions. When studying cross-modal interactions, multimodal data alignment is difficult, and the reasoning ability of the model will be limited. Too many hierarchical learning levels will cause information loss or overfitting problems, increase the time cost of model training, and reduce the efficiency of actual applications. Modular networks and symbolic reasoning have limited performance when dealing with problems involving multimodal long-term dependency reasoning, and have poor adaptability to free question-answering scenarios in open domains; the present invention proposes an autonomous driving video question-answering method, a model training method and a system, wherein a training sample set trains an initial autonomous driving question-answering model, and each sample contains continuous video image data, audio data and text data of questions raised for a scene in an autonomous driving video generated for an autonomous driving video; the video features and audio features extracted by the extraction module are fused after feature compression to obtain audio-video fusion features, which are input into the scene hypergraph generation module, the relationship embedding layer obtains the relationship query input, the relationship decoder obtains the relationship features, and the relationship classification head outputs the relationship classification results; the driving behavior embedding layer obtains the driving behavior query input, the driving behavior decoder obtains the driving behavior features , the driving behavior classification head outputs the driving behavior classification result; the scene hypergraph embedding layer adds the embedding of the corresponding relationship classification result and time step to the relationship feature and the driving behavior feature respectively, concatenates and introduces an identity embedding to obtain the scene hypergraph embedding; the question text feature is compressed and input into the video question answering module together with the scene hypergraph embedding, and the answer prediction value about the question text data is output; the Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The cross entropy loss function is constructed according to the deviation between the true value of the answer and the predicted answer value. The fusion loss function is constructed through the two loss functions. The parameters of the initial autonomous driving video question answering model are iteratively updated with the goal of minimizing the fusion loss function to obtain the autonomous driving video question answering model.

[0053] Figure 1 The figure is a flowchart of a method for training an autonomous driving video question-answering model according to an embodiment of the present invention. Figure 2 Schematic diagram of the structure of the autonomous driving video question-answering model according to an embodiment of the present invention. Specifically, the present application provides a method for training an autonomous driving video question-answering model, which includes the following steps S101 to S103:

[0054] Step S101: Obtain a training sample set, where the training sample set includes multiple samples, each sample includes continuous video image data, audio data, and question text data generated for an autonomous driving video, with the true value of the driving state relationship between objects in the autonomous driving video, the true value of the driving behavior category, and the true value of the answer to the question text data as labels.

[0055] Step S102: The initial autonomous driving question-answering model is trained using a training sample set, wherein the initial autonomous driving question-answering model comprises: a video feature extraction module for extracting video features from video image data, an audio feature extraction module for extracting audio features from audio data, a question text feature extraction module for extracting question text features from question text data, a video-audio feature fusion module, a scene hypergraph generation module and a video question-answering module; the video features and the audio features are compressed and then fused by the video-audio feature fusion module to obtain audio-video fusion features; the scene hypergraph generation module comprises a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head and a scene hypergraph embedding layer; the relationship embedding layer obtains relationship queries from the audio-video fusion features, and the relationship decoder is based on the relationship query. The cross attention mechanism obtains relational features from relational queries and audio and video fusion features, and then the relation classification head outputs the relational classification results; the driving behavior embedding layer obtains driving behavior queries from audio and video fusion features, and the driving behavior decoder obtains driving behavior features from driving behavior queries and audio and video fusion features based on the cross attention mechanism, and then the driving behavior classification head outputs the driving behavior classification results; the scene hypergraph embedding layer adds the embedding corresponding to the relational classification results and time steps to the relational features, and adds the embedding corresponding to the driving behavior classification results and time steps to the driving behavior features, and then the scene hypergraph embedding is obtained by splicing; the scene hypergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video; the question text features are compressed and input into the video question-answering module together with the scene hypergraph embedding to output the answer prediction value about the question text data.

[0056] Step S103: construct a Hungarian matching loss function according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result; construct a cross entropy loss function according to the deviation between the true value of the answer and the predicted answer value; obtain a fusion loss function through the Hungarian matching loss function and the cross entropy loss function; and iteratively update the parameters of the initial autonomous driving video question and answer model with the goal of minimizing the fusion loss function to obtain an autonomous driving video question and answer model.

[0057] In step S101, the continuous video image data of each sample in the training set is a set number of continuous pictures with the same interval obtained after resizing the pictures in the same autonomous driving video. The audio data in each sample is the audio of an autonomous driving video evenly divided into a set number of audio clips of the same length. The question text data in each sample is the question corresponding to an autonomous driving video.

[0058] Furthermore, the true value of the driving behavior category in each label and the true value of the answer to the question text data are obtained from a preset data set, and the driving state relationship between the objects is obtained by a pre-trained target detection network and a scene graph generation network; in some embodiments, the step of obtaining the true value of the driving state relationship between the objects in the autonomous driving video includes steps S1011 to S1012:

[0059] Step S1011: The preprocessed continuous video image data is input into a pre-trained object detection network to obtain the bounding box information and object type of each object in each video image.

[0060] Step S1012: The pre-trained scene graph generation network generates a scene graph of continuous video image data through the bounding box information and object type of each object and obtains the object category in each video image and the true value of the driving state relationship between each object.

[0061] Specifically, the preprocessing of continuous video image data is to transform the dimension of the continuous video image data to adapt to the input of the subsequent video feature extraction module; the target detection network includes a region-based convolutional neural network (R-CNN) series and a YOLO series, which are used to detect objects in the input continuous video images, and each video image contains a vehicle representing the current autonomous driving; the scene graph generation network includes a spatial decoder and a temporal decoder, which generate a scene graph of each video image according to the spatial information and temporal information between each object in each video image. The spatial information includes the position, shape, size of each object and the relationship between each object, and the temporal information includes the change of each object on the timeline and the interaction with other objects during the change process. The driving state relationships between objects include the front-and-rear, parallel and cross driving position relationships between vehicles or between vehicles and other objects, the relative speed relationships between vehicles or between vehicles and other objects in the same direction and speed, the same direction and different speeds and in opposite directions, and the distance relationships between vehicles or between vehicles and other objects; the driving behavior categories include the speed changing behavior of vehicles or other objects, the turning behavior of vehicles or other objects, and the stopping behavior of vehicles or other objects; the answers to the question text data include 26 answers to all the questions in the preset data set.

[0062] In step S102, in some embodiments, the video feature extraction module includes a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module includes a spectrogram conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module includes an embedding layer, two layers of recurrent neural networks, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module includes three layers of self-attention layers and two fully connected layers; the video question-answering module includes multiple layers of cross-attention layers, a linear layer and a softmax function layer.

[0063] Specifically, the multi-layer convolutional network contains five convolutional layers and two pooling layers. The first three convolutional layers are used to extract low-level features, including edges and textures, and the last two convolutional layers are used to extract high-level spatiotemporal information, including motion patterns and object relationships. In the process of video feature extraction, the features are obtained through the multi-layer convolutional network, flattened, and then compressed by linear layers to obtain video features; in the process of audio feature extraction, each audio segment in the audio data is represented as a spectrum graph, and the features are obtained through the multi-layer convolutional network and then compressed by linear layers to obtain audio features; in the process of question text feature extraction, the question text data is word segmented and embedded, and then the features are obtained through a two-layer recurrent neural network. The time step of the feature is pooled to compress the length, and then flattened and compressed by linear layers to obtain question text features. The question text features are compressed by three self-attention layers to input into the video question answering module.

[0064] In some embodiments, when the video features and the audio features are feature compressed, the three self-attention layers of the audio-visual feature fusion module are used, including steps S1021 to S1023:

[0065] Step S1021: The first self-attention layer obtains a first query matrix, a first key matrix and a first value matrix through three different linear transformations on the input features, and obtains the first output features according to the self-attention calculation formula.

[0066] Step S1022: The second self-attention layer obtains a second query matrix, a second key matrix and a second value matrix for the first output feature through three different linear transformations, and obtains the second output feature according to the self-attention calculation formula.

[0067] Step S1023: The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix for the second output feature through three different linear transformations, and obtains the third output feature according to the self-attention calculation formula.

[0068] Specifically, the video features and audio features are input into the audio-video feature fusion module and then compressed by three self-attention layers. Then, they are fused by two fully connected layers in the audio-visual feature fusion module to obtain the audio-video fusion features. Each fully connected layer contains two linear layers and one Relu activation layer.

[0069] Furthermore, the relation embedding generation layer includes a relation embedding layer and a driving behavior embedding layer, the audio and video fusion feature input scene hypergraph generation module obtains relation queries through the relation embedding layer and obtains driving behavior queries through the driving behavior embedding layer respectively, the relation decoder obtains relation features based on the relation query and the audio and video fusion features and the relation classification head outputs the relation classification results, the driving behavior decoder obtains driving behavior features based on the driving behavior query and the audio and video fusion features and the driving behavior classification head outputs the driving behavior classification results; the relation decoder and the driving behavior decoder include multiple layers of cross attention layers, and the relation classification head and the driving behavior classification head include two fully connected layers. The scene hypergraph embedding layer adds the embedding corresponding to the relation classification results and time steps, the embedding corresponding to the driving behavior classification results and time steps, and an identification embedding representing the overall content of the autonomous driving video, which is used for the scene hypergraph generation module to learn the relationship between features in different categories and different time steps, and the embedding in the scene hypergraph embedding layer is provided by a PyTorch framework containing a set number of learnable embeddings; the identification embedding is used to represent the overall content in the autonomous driving video, including but not limited to the driving environment, surrounding weather, road traffic conditions and vehicle status in the current autonomous driving video.

[0070] In some embodiments, the expression of the relation decoder based on the cross attention mechanism is:

[0071] Q=R q W q , K = FW k , V=FW v ;

[0072]

[0073] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents a relational query, K represents the key matrix, and W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function.

[0074] The expression of the driving behavior decoder based on the cross attention mechanism is:

[0075] Q=Aq W q , K = FW k , V=FW v ;

[0076]

[0077] Among them, A q Represents a driving behavior query.

[0078] Furthermore, the video question answering module includes multiple cross attention layers, a linear layer and a softmax function layer, and the linear layer includes two fully connected layers; in some embodiments, the question text features are compressed and embedded with the scene hypergraph before being input into the video question answering module to output the answer prediction value about the question text data, including steps S1 to S2:

[0079] Step S1: The multi-layer cross-attention layer in the video question-answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs them into the linear layer to obtain the answer classification embedding.

[0080] Step S2: The softmax function layer obtains the probability of each answer to the question text data through answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

[0081] Specifically, the multi-layer cross attention layer performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression and then inputs them into the linear layer to obtain the answer classification embedding A:

[0082] Q=TW q , K = HW k , V = HW v ;

[0083]

[0084] A=Linear(Attention(Q,K,V));

[0085] Among them, T represents the question text features and H represents the scene hypergraph embedding.

[0086] In step S103, the Hungarian matching loss function is used to train the scene hypergraph generation module to improve the accuracy of the generated scene hypergraph. The Hungarian matching loss function includes relationship loss and driving behavior loss. The cross entropy loss function is used to train the video question answering module to improve the accuracy of the video question answering task.

[0087] In some embodiments, a Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result, and the expression is:

[0088] L1=λ1L rel +λ2L act ;

[0089]

[0090]

[0091] Among them, L rel represents the relational loss formed by the deviation between the true value of the driving state relation and the relational classification result, λ1 represents the hyperparameter of the relational loss, r represents the true value of the driving state relation, represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss composed of the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

[0092] In some embodiments, a cross entropy loss function is constructed based on the deviation between the true answer value and the predicted answer value, and the expression is:

[0093]

[0094] Among them, yi represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

[0095] Another invention, the present invention also provides an autonomous driving video question-answering method, the method comprising steps S201-S202:

[0096] Step S201: Obtain the autonomous driving video to be detected and the question text posed for the autonomous driving video.

[0097] Step S202: Input the autonomous driving video to be detected and the question text into the autonomous driving video question and answer model obtained by any of the above autonomous driving video question and answer model training methods, and output the answer to the question in the autonomous driving video.

[0098] On the other hand, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any one of the above methods.

[0099] The present invention is described below in conjunction with a specific embodiment:

[0100] 1. Obtain training sample sets and labels.

[0101] Resize each video image in each autonomous driving video to obtain F consecutive images P = {P1, P2, ..., P F}, each video image is preprocessed and converted into a picture p = {p1, p2, ..., p F}, similarly, the audio of each autonomous driving video is evenly divided into F audio clips of the same length and no less than 1.2 seconds as the input of the feature extraction module.

[0102] First, each video image is fed into a pre-trained object detection network to obtain the bounding box (1×5) and object category label (1×N) of each object in each video image, where N is the number of object categories, and each video image contains the current autonomous driving vehicle in the center of the picture. Then these objects are fed into a pre-trained scene graph generation network, which mainly includes a spatial encoder and a temporal decoder to generate a scene graph from each video image. The object category (M×N) in each video image and the driving state relationship between each object are obtained, and the driving state relationship label and the driving behavior category label in the dataset are integrated to obtain the overall label (C×R), where M is the number of objects in each video image, C is the number of object pairs in the current video image, and R is the sum of all driving behaviors and driving state relationship categories. At the same time, the question corresponding to each autonomous driving video and the corresponding correct answer are obtained for calculating the loss.

[0103] 2. Build an autonomous driving video question-answering system.

[0104] 2.1 Construct the video feature extraction module, audio feature extraction module, question text feature extraction module and audio-visual feature fusion module. The video feature extraction module is a multi-layer convolutional network with five convolutional layers and two pooling layers. The first three layers are responsible for extracting low-level features, including but not limited to edges and textures, and the last two layers capture high-level spatiotemporal information through deeper convolutional structures, including but not limited to motion patterns and object relationships. The audio feature extraction module first converts each audio segment into a spectrogram to capture frequency domain features, and then uses a multi-layer convolutional network with five convolutional layers and two pooling layers. The feature extraction module effectively extracts important information from video and audio. The question text feature extraction module first performs word segmentation and embedding processing, converts each question text into a word embedding representation, and then extracts features through a two-layer recurrent neural network structure to capture contextual information in the question text. The output question text features will be pooled to reduce the time step. The output video features, audio features and question text features will be dimensionally compressed through a linear layer to ensure that the feature dimensions remain consistent for subsequent fusion and processing. Each modality feature uses a unimodal encoder, which consists of three self-attention layers, a residual network, and a linear layer.

[0105] The audio-visual feature fusion module consists of three self-attention layers and two fully connected layers, each of which contains two linear layers and one ReLU activation layer.

[0106] 2.2 Construct the scene hypergraph generation module. The scene hypergraph generation module contains two relation generation networks, each of which contains a relation query embedding generation layer, a decoder and a classification head. The relation decoder consists of multiple layers of cross attention layers, and the relation classification head consists of two fully connected layers. The relation query embedding generation layer includes a relation embedding layer and a driving behavior embedding layer, the relation decoder includes a relation decoder and a driving behavior decoder, and the relation classification head includes a relation classification head and a driving behavior classification head.

[0107] 2.3 Constructing the video question answering module. The video question answering module consists of a cross-attention module, which contains multiple cross-attention layers, followed by two fully connected layer modules and a softmax function layer.

[0108] 3. The video feature extraction module extracts video features, the audio feature extraction module extracts audio features, the question text feature extraction module extracts question text features, and the audio-visual feature fusion module fuses video features and audio features. For video data, the first input is video image data (16×3×224×224), and the video image data is feature extracted through a multi-layer convolutional network to obtain video features (16×7×7×16). Since the subsequent attention network needs to process sequence data, the video features are first flattened to obtain features (16×49×16), and the features are compressed to 16×1024 through a linear layer. For audio features, each audio segment is first represented as a spectrogram, which is processed through a multi-layer convolutional network to obtain features (16×256×128), and finally compressed through a linear layer to obtain audio features (16×1024); for text features, each question text is first segmented and embedded to obtain the initial word embedding representation (128×300), and then processed through the question text feature extraction network to obtain question text features (128×512). In order to unify the processing method with video and audio features, the time step of the question text feature is pooled and its length is compressed to 64. Then, the question text feature is flattened and its dimension is compressed through a linear layer to obtain text features (16×512).

[0109] Next, the three modal features are processed by three self-attention layers respectively, which helps the audio and video fusion features (16×1024) to further learn the relationship of time context. In order to obtain the fusion features of video and audio, the video features and audio features are concatenated and then passed through a linear layer to obtain the audio and video fusion features F (16×1024).

[0110] 4. Input the audio and video fusion features into the scene hypergraph generation module and output the scene hypergraph embedding corresponding to the autonomous driving video. First, the corresponding relationship query R is obtained through the relationship embedding layer and the driving behavior embedding layer. q (128×512) and driving behavior query A q (48×512), the relationship query and driving behavior query are respectively input into the respective relationship decoder and driving behavior decoder together with the audio and video fusion features of the previous step, and the relationship feature R representing each frame scene is obtained through multi-layer cross attention f (129×512) and driving behavior characteristics A f (48×512), taking relation query as an example, the calculation of cross attention is as follows:

[0111] Q=R q W q ,K=FW k ,V=FWv ;

[0112]

[0113] Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents a relational query, K represents the key matrix, and W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function.

[0114] The relationship features and driving behavior features are respectively passed through the relationship classification head and the driving behavior classification head to obtain the classification results of each relationship: relationship classification results (128×418) and driving behavior classification results (48×48).

[0115] Next, we add the embedding of the corresponding relationship classification results and time steps to the relationship features, and add the embedding of the corresponding driving behavior classification results and time steps to the driving behavior features, learn the relationships at different time steps, and concatenate them to get the scene hypergraph embedding representing the entire video. We then add a representation embedding [HG](1×512) representing the scene hypergraph type to this scene hypergraph embedding to get the final scene hypergraph embedding H(177×512):

[0116] H=R s +A s +[HG];

[0117] 5. Input the scene hypergraph embedding and question text features into the video question answering network and output the final answer to the question. Input the scene hypergraph embedding and text features T into the video question answering network through a multi-layer cross-attention network. The calculation of the cross-attention layer is as follows:

[0118] Q=TW q , K = HW k , V = HW v ;

[0119]

[0120] Among them, H is the scene hypergraph, T is the text feature, and then this output is passed through the linear layer to get the answer classification embedding A:

[0121] A=Linear(Attention(Q,K,V));

[0122] The answer is embedded in the softmax function to obtain the probability of each answer in the data set. The one with the highest probability is the predicted answer value of the question text data.

[0123] 6. Construct a loss function. In order to train the scene hypergraph generation module's scene hypergraph generation capability, the Hungarian matching loss function is used. The goal of this loss function is to enable the scene hypergraph generation module to have the ability to correctly generate scene hypergraphs, so as to improve the accuracy of using the scene hypergraph embedding for video question-answering tasks in subsequent stages. The calculation formula of the loss function is as follows:

[0124]

[0125] Among them, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, The predicted probability of the classification result. For the relationship set loss function L rel It is similar to the driving behavior set loss function, where r represents the true value of the driving state relationship. Represents the relationship classification result, and M represents the size of the relationship set:

[0126]

[0127] Next, we use a cross entropy loss function. The goal of this loss function is to train the video question-answering module to correctly answer questions, so that the video question-answering module gives a higher probability of the final classification of the correct answer. The expression of this loss function is:

[0128]

[0129] Among them, y i Represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

[0130] The expression of the fusion loss function is:

[0131] L=λ1L act +λ2L rel +L vqa

[0132] Among them, λ1 represents the hyperparameter of the relationship loss, and λ2 represents the hyperparameter of the driving behavior loss.

[0133] 7. Input the test set into the trained model to get the answer to each question. Resize the video images in the autonomous driving videos in the test set to obtain video image data, obtain the corresponding audio data and the question text data corresponding to each video, and then input the processed features into the trained autonomous driving video question-answering model to get the answer to the autonomous driving video question.

[0134] In summary, the present invention provides an autonomous driving video question-answering method, a model training method and a system. Each sample in the training sample set includes continuous video image data, audio data and question text data for a scene in the autonomous driving video generated for an autonomous driving video; the video features and audio features extracted by the feature extraction module are fused after feature compression to obtain audio-video fusion features, and are input into the scene hypergraph generation module; the relationship embedding layer obtains the relationship query input, the relationship decoder obtains the relationship features, and the relationship classification head outputs the relationship classification results; the driving behavior embedding layer obtains the driving behavior query input, the driving behavior decoder obtains the driving behavior features, and the driving behavior classification head outputs the driving behavior classification results; the scene The scene hypergraph embedding layer adds the embedding of corresponding relationship classification results and time steps to the relationship features and driving behavior features respectively, concatenates and introduces a logo embedding to obtain the scene hypergraph embedding; the question text features are compressed and input into the video question answering module together with the scene hypergraph embedding to output the answer prediction value for the question text data; the Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. The cross entropy loss function is constructed according to the deviation between the true value of the answer and the predicted answer value. The fusion loss function is constructed by the two loss functions to iteratively update the parameters of the initial autonomous driving video question answering model to obtain the autonomous driving video question answering model.

[0135] Corresponding to the above method, the present invention also provides an autonomous driving video question-and-answer system, comprising a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described above.

[0136] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0137] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the key shielding method of the vehicle-mounted display device in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0138] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The one or more modules are stored in the memory, and when executed by the processor, the method described in this embodiment is performed.

[0140] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0141] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0142] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0143] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training an autonomous driving video question-answering model, characterized in that: The method comprises the following steps: Obtain a training sample set, wherein the training sample set includes a plurality of samples, each sample includes continuous video image data, audio data, and question text data generated for a scene in the autonomous driving video, and uses the true value of the driving state relationship between objects in the autonomous driving video, the true value of the driving behavior category, and the true value of the answer to the question text data as labels; The training sample set is used to train an initial autonomous driving question-answering model, and the initial autonomous driving question-answering model includes: a video feature extraction module for extracting video features from the video image data, an audio feature extraction module for extracting audio features from the audio data, a question text feature extraction module for extracting question text features from the question text data, a visual and audio feature fusion module, a scene hypergraph generation module, and a video question-answering module; the video features and the audio features are fused by the visual and audio feature fusion module after feature compression to obtain audio-video fusion features; the scene hypergraph generation module includes a relationship embedding layer, a relationship decoder, a relationship classification head, a driving behavior embedding layer, a driving behavior decoder, a driving behavior classification head, and a scene hypergraph embedding layer; the relationship embedding layer obtains a relationship query from the audio-video fusion feature, and the relationship decoder performs a cross-attention on the relationship query based on the relationship query. The driving behavior embedding layer obtains a driving behavior query from the audio and video fusion features, and the driving behavior decoder obtains a driving behavior feature from the driving behavior query and the audio and video fusion features based on a cross attention mechanism, and the driving behavior classification head outputs a driving behavior classification result; the scene hypergraph embedding layer adds an embedding of the corresponding relationship classification result and time step to the relationship feature, and adds an embedding of the corresponding driving behavior classification result and time step to the driving behavior feature, and splices them to obtain a scene hypergraph embedding; the scene hypergraph embedding introduces an identifier embedding representing the overall content of the autonomous driving video; the question text feature is compressed and input into the video question-answering module together with the scene hypergraph embedding to output a predicted value of the answer to the question text data; A Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result. A cross entropy loss function is constructed according to the deviation between the true value of the answer and the predicted answer value. A fusion loss function is obtained by using the Hungarian matching loss function and the cross entropy loss function. The parameters of the initial autonomous driving video question and answer model are iteratively updated with the goal of minimizing the fusion loss function to obtain an autonomous driving video question and answer model.

2. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The step of obtaining the true value of the driving state relationship between the objects in the autonomous driving video includes: The pre-processed continuous video image data is input into a pre-trained object detection network to obtain the bounding box information and object type of each object in each video image; The pre-trained scene graph generation network generates a scene graph of the continuous video data through the bounding box information of each object and the object type and obtains the object category in each video image and the true value of the driving state relationship between each object.

3. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The video feature extraction module includes a multi-layer convolutional network, a flattening processing layer and a linear layer; the audio feature extraction module includes a spectrum graph conversion layer, a multi-layer convolutional network and a linear layer; the question text feature extraction module includes an embedding layer, a two-layer recurrent neural network, a pooling layer, a flattening processing layer, a linear layer and a three-layer self-attention layer; the audio-visual feature fusion module includes three layers of self-attention layers and two fully connected layers; the video question-answering module includes multiple layers of cross-attention layers, a linear layer and a softmax function layer.

4. The autonomous driving video question-answering model training method according to claim 2, characterized in that: The video features and the audio features are feature compressed using the three self-attention layers of the audio-visual feature fusion module, including: The first self-attention layer obtains the first query matrix, the first key matrix and the first value matrix through three different linear transformations on the input features, and obtains the first output feature according to the self-attention calculation formula; The second self-attention layer obtains a second query matrix, a second key matrix and a second value matrix by performing three different linear transformations on the first output feature, and obtains a second output feature according to the self-attention calculation formula; The third self-attention layer obtains a third query matrix, a third key matrix and a third value matrix for the second output feature through three different linear transformations, and obtains the third output feature according to the self-attention calculation formula.

5. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The expression of the relation decoder based on the cross attention mechanism is: Q=R q W q ,K=FW k ,V=FW v ; Where Q represents the query matrix, W q represents the weight of the query matrix, R q represents the relational query, K represents the key matrix, W k represents the weight of the key matrix, F represents the audio and video fusion feature, V represents the value matrix, W v represents the weight of the value matrix, d represents the dimension of the input tensor, QK T is the dot product matrix of query and key, and softmax is the activation function; The driving behavior decoder is expressed based on the cross attention mechanism as follows: Q=A q W q ,K=FW k ,V=FW v ; Among them, A q represents the driving behavior query.

6. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The question text features are compressed and embedded in the scene hypergraph before being input into the video question-answering module to output the answer prediction value about the question text data, including: The multi-layer cross attention layer in the video question answering module performs weighted fusion on the scene hypergraph embedding and the question text features after feature compression, and then inputs the result into the linear layer to obtain the answer classification embedding; The softmax function layer obtains the probability of each answer to the question text data through the answer classification embedding, and selects the answer with the largest output probability as the answer prediction value.

7. The autonomous driving video question-answering model training method according to claim 1, characterized in that: A Hungarian matching loss function is constructed according to the deviation between the true value of the driving state relationship and the relationship classification result, and the deviation between the true value of the driving behavior category and the driving behavior classification result, and the expression is: L1=λ1Lrel+λ2Lact; Among them, L rel represents the relation loss formed by the deviation between the true value of the driving state relationship and the relationship classification result, λ1 represents the hyperparameter of the relation loss, r represents the true value of the driving state relationship, represents the relationship classification result, M represents the size of the relationship set, L act represents the driving behavior loss formed by the deviation between the true value of the driving behavior category and the driving behavior classification result, λ2 represents the hyperparameter of the driving behavior loss, a represents the true value of the driving behavior category, represents the driving behavior classification result, N represents the size of the driving behavior set, T represents the time step, Represents the predicted probability of the classification result.

8. The autonomous driving video question-answering model training method according to claim 1, characterized in that: The cross entropy loss function is constructed based on the deviation between the true value of the answer and the predicted value of the answer, and the expression is: Among them, y i represents the true value of the answer to the i-th question text data, represents the predicted answer value of the i-th question text data, and B represents the total number of questions.

9. An autonomous driving video question-answering method, characterized in that: The method comprises: Obtaining an autonomous driving video to be detected and a text of a question raised with respect to the autonomous driving video; The autonomous driving video to be detected and the question text are input into the autonomous driving video question and answer model obtained by the autonomous driving video question and answer model training method according to any one of claims 1 to 8, and the answer to the question in the autonomous driving video is output.

10. An autonomous driving video question-answering system, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 9.

Images