A video question-answering method and system with explainability and knowledge elicitation capability

Abstract

The application discloses a video question and answer method and system with explainability and knowledge enlightenment capability. The method comprises the following steps: inputting a video into a VGG network and an I3D network to extract a video feature sequence; inputting a user question into a BERT network to extract a question vector representation; inputting the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of a video scene; inputting the question vector representation and the causal representation of the video scene into a GPT-3 model to generate a knowledge-guided question representation through the coding-decoding structure of the model; inputting the knowledge-guided question representation into a Transformer decoding network to obtain an answer to the user question; and visualizing the knowledge-guided question representation and the answer to the user question through a self-attention mechanism, a visualization tool and an explanatory description. The application can improve the accuracy, explainability and robustness of a video question and answer model, thereby improving user experience and the practicality of application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video understanding, and in particular to a video question answering method and system with explainability and knowledge elicitation capability. BACKGROUND

[0002] Video question answering provides a new way for search, browsing and understanding of video content, and has a wide application space in the current, including intelligent assistants, education and training, video content management and other fields. How to realize the cross-modal understanding and alignment between vision and language and solve the fusion and reasoning of multi-modal information is a key technical challenge in the field of video question answering. At the same time, the explainability of the reasoning process and the generated results is also one of the important indicators for evaluating video question answering models.

[0003] Causal reasoning can enable video question answering models to more accurately understand the causal relationships between different elements in the video, identify causal scenarios that are critical to answer generation, and provide more accurate answers. There are various ways to realize the causal relationship modeling of video question answering models, including traditional statistical causal models such as causal graphs, causal Bayesian networks, and deep learning models.

[0004] In the field of video question answering, large language models can be used as a base model to provide key functions such as text understanding, question answering and knowledge reasoning. Through pre-training, large language models can learn general language knowledge and semantic representations in massive amounts of text data, and on this basis, extract key information from questions, generate accurate and coherent answers, and guide and supervise video question answering models.

[0005] By introducing external knowledge to guide video question answering models, it is helpful to more accurately understand the question, better combine the question and video content, conduct more detailed reasoning, and generate more accurate answers.

[0006] One of the current state-of-the-art methods is the paper "KnowIT VQA: Answering knowledge-based questions about videos", which combines information from video content and external knowledge base, utilizes self-attention mechanism for multi-modal fusion and reasoning, to generate accurate answers. First, the video frames and questions are transformed into low-dimensional embedding representations through encoding. Then, the self-attention mechanism is used to dynamically capture the relevance between video and question, and extract key information from video frames. In addition, the knowledge of the external knowledge base is used as additional input, which is associated with video content and questions by encoding and applying attention mechanism on the knowledge. During the training phase, the model is trained by maximizing the probability of the correct answer, using cross-entropy loss function for optimization. Through large-scale video question answering dataset, the model can learn the complex relationship between video and question, and has the ability to accurately answer knowledge-based questions. The disadvantage of this method is the high dependence on external knowledge base and the high demand for computing resources.

[0007] Another current state-of-the-art method is the paper "Knowledge-based video question answering with unsupervised scene descriptions", which combines information from video content and unsupervised scene descriptions, and realizes the goal of accurate question answering by fusing visual features and knowledge representation. In this method, the video is represented as a series of frames, and each frame is extracted by a pre-trained visual feature extractor. At the same time, unsupervised scene description technology is used to automatically generate descriptions of video frames. These scene descriptions provide information related to video content as part of the external knowledge base. When answering questions, first, the question and the visual features of the video frames are encoded into low-dimensional vector representations. Then, by calculating the similarity between the question and the video frames, the most relevant video frames to the question are selected. Next, the knowledge representation and attention mechanism are used to integrate the information in the external knowledge base into the question answering process. Finally, according to the fused information, an accurate answer is generated. The disadvantage of this method is that the quality of unsupervised scene description cannot be guaranteed, which affects the accuracy of question answering, and is limited by the quality and coverage of the external knowledge base.

[0008] The third prior art is the paper "iPerceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering", which combines video content and common-sense knowledge to achieve accurate video caption generation and question answering through multi-modal fusion and common-sense reasoning mechanism. In this method, the video is represented as multi-modal data, including video frames, audio and text. First, the feature representations of video frames and audio are extracted by pre-trained visual and audio feature extractors. Then, text processing techniques are used to generate descriptions of video frames and representations of questions. When generating video captions, visual, audio and text features are fused, and common-sense reasoning mechanism is applied to introduce external common-sense knowledge into the caption generation process. This can improve the accuracy and semantic consistency of the captions. When answering questions, the question is matched with the features of the video frames, and the most relevant video frames to the question are selected. Then, the common-sense reasoning mechanism is used to reason and select answers to the question, providing accurate answers to the question. The disadvantage of this method is that the accuracy of the model output is limited by the quality and coverage of external common-sense knowledge, and the model decision-making process and reasoning steps lack explainability. SUMMARY

[0009] The purpose of the present application is to overcome the shortcomings of existing methods and propose a video question answering method and system with explainability and knowledge elicitation capability. The main problem solved by the present application is to improve the understanding and answering ability of difficult questions, reduce the dependence of the video question answering model on external knowledge, and enhance the explainability of the output answers and reasoning process, thereby improving the accuracy, explainability and robustness of the video question answering model.

[0010] To solve the above problems, the present application proposes a video question answering method with explainability and knowledge elicitation capability, which comprises:

[0011] inputting the video into a VGG network and an I3D network to extract a video feature sequence;

[0012] inputting the user question into a BERT network to extract a question vector representation;

[0013] inputting the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of the video scene;

[0014] inputting the question vector representation and the causal representation of the video scene into a GPT-3 model to generate a knowledge-guided question representation through the encoding-decoding structure of the model;

[0015] inputting the knowledge guided question representation into a Transformer decoding network to obtain an answer to the user question;

[0016] The knowledge guided question representation and the answer to the user question are visualized as an answer selected and a reasonable explanation generated for the user to understand through a self-attention mechanism, a visualization tool and an explanatory description.

[0017] Preferably, a video input VGG network and an I3D network extract a video feature sequence, specifically:

[0018] The video feature sequence includes a static feature representation of a video frame and a dynamic feature representation of a video, and the extraction method is as follows:

[0019] The video is input into a pre-trained VGG network in units of frames, and the local features and global features of the video frame image are extracted layer by layer through the convolutional layers and pooling layers in the VGG network to obtain a static feature representation of each video frame;

[0020] The video is input into a pre-trained I3D network, and the dynamic changes of the video are modeled in the time domain through the time dimension convolution operation of the convolutional layers in the I3D network to obtain a dynamic feature representation of the video;

[0021] The static feature representation of the video frame and the dynamic feature representation of the video are fused by using a cross-Transformer network to obtain a video feature sequence.

[0022] Preferably, a user question is input into a BERT network to extract a question vector representation, specifically:

[0023] The user question is preprocessed, that is, the user question is subjected to natural language processing operations of word segmentation, part-of-speech tagging and syntax analysis to obtain a preprocessed question;

[0024] The preprocessed question is converted into a question vector representation with semantic information by the BERT network.

[0025] Preferably, the video feature sequence is input into a Bayesian differentiable neural network and a cross-Transformer network to obtain a causal representation of a video scene, specifically:

[0026] Let the video feature sequence X = {X1, X2, …, X n}, where X i , is the video feature of the i-th frame of the video, and the video feature sequence X is input into the cross-Transformer network to obtain visual features related to the user question by using an attention mechanism;

[0027] input the visual feature into the cross Transformer network with the problem vector representation to obtain a problem-guided video feature X i q ;

[0028] input the problem-guided video feature X i q into a Bayesian differentiable neural network, and train and optimize the Bayesian differentiable neural network through a back propagation algorithm to obtain P(X i |X i-1 ,X i-2 ,…,X1), that is, a probability of occurrence of the i-th frame video feature under a condition of given video features of the 1st frame to the i-1th frame;

[0029] A specific calculation formula of the causal reasoning is as follows:

[0030]

[0031] wherein ε is a random variable of a Gaussian sampling distribution I k is a covariance matrix with a sampling number k, and is a parameterized representation of the encoding network which can be learned, is a parameterized representation of the decoding network which can be learned, x u is a feature of X i processed by the cross Transformer network, and z u is a video feature output which is mapped by the learnable parameters.

[0032] Preferably, the knowledge-guided problem representation is input into a Transformer decoding network to obtain an answer to the user question, specifically as follows:

[0033] The knowledge-guided problem representation is input into the Transformer decoding network, and a generated answer to the question is decoded by using a self-attention mechanism and a feedforward neural network layer of the Transformer decoding network.

[0034] The generated answer to the question is compared with a standard answer to the question by using a cross-entropy loss function, and the generated answer to the question is optimized by minimizing the cross-entropy loss function to obtain a more accurate answer to the question.

[0035] The video feature representation is input into a GPT-3 model, and the generated answer is combined as a supervision signal to supervise training of the Transformer network.

[0036] Preferably, the knowledge guided question representation and the answer to the user question are visualized as reasonable explanations for the answer selection and generation that are easy for the user to understand by using self-attention mechanism, visualization tools and explanatory descriptions, specifically:

[0037] attention weights when calculating and generating the answer by using the self-attention mechanism;

[0038] the attention weights, the video feature sequence and the question vector representation are inputted by using Matplotlib, Plotly and D3.js to generate charts and images, which assist the user to intuitively understand the basis and reasoning process of the answer generation;

[0039] the attention weights are inputted by using the explanatory description to generate the analysis of the question, the supporting evidence for the answer generation and the explanation of the reasoning path.

[0040] Correspondingly, the application also provides a video question answering system with explainability and knowledge guiding capability, comprising:

[0041] a video encoding unit configured to input a video into a VGG network and an I3D network to extract a video feature sequence;

[0042] a text encoding unit configured to input a user question into a BERT network to extract a question vector representation;

[0043] a causal scenario modeling unit configured to input the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of a video scenario;

[0044] a knowledge introduction unit configured to input the question vector representation and the causal representation of the video scenario into a GPT-3 model to generate a knowledge guided question representation by using the encoding-decoding structure of the model;

[0045] an answer generation unit configured to input the knowledge guided question representation into a Transformer decoding network to obtain an answer to the user question;

[0046] a visualization unit configured to visualize the knowledge guided question representation and the answer to the user question as reasonable explanations for the answer selection and generation that are easy for the user to understand by using self-attention mechanism, visualization tools and explanatory descriptions.

[0047] The application has the following beneficial effects:

[0048] The application provides a video question answering method with explainability and knowledge enlightenment capability. The application introduces scene modeling of causal reasoning, a large language model and a knowledge enlightenment method, so that the video question answering model can better understand the problem and the causal relationship between the video content and details in the video, and reduce the dependence of the video question answering model on external knowledge, thereby improving the understanding of difficult problems and the accuracy of generated answers. Unlike other prior art, the application also emphasizes the explanation and display of the output results and the reasoning process, which enhances the user's trust in the model output results, facilitates the user's supervision and improvement of the model, and significantly improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of a video question answering method with explainability and knowledge enlightenment capability according to an embodiment of the application;

[0050] Figure 2 is a causal reasoning diagram according to an embodiment of the application;

[0051] Figure 3 is a structural diagram of a video question answering system with explainability and knowledge enlightenment capability according to an embodiment of the application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0053] Figure 1 is a flowchart of a video question answering method with explainability and knowledge enlightenment capability according to an embodiment of the application, as shown in Figure 1 , the method comprises:

[0054] S1, inputting a video into a VGG network and an I3D network to extract a video feature sequence;

[0055] S2, inputting a user question into a BERT network to extract a question vector representation;

[0056] S3, inputting the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of a video scene;

[0057] S4, inputting the question vector representation and the causal representation of the video scene into a GPT-3 model to generate a knowledge-guided question representation through the encoding-decoding structure of the model;

[0058] S5, inputting the knowledge guided question representation into a Transformer decoding network to obtain an answer to the user question;

[0059] S6, visualizing the knowledge guided question representation and the answer to the user question into an answer selection and a reasonable explanation generated by a self-attention mechanism, a visualization tool and an explanatory description, which are easy for the user to understand.

[0060] Step S1 is specifically as follows:

[0061] S1-1, the video feature sequence includes a static feature representation of a video frame and a dynamic feature representation of a video, and the extraction method is as follows:

[0062] The video is input into a pre-trained VGG network, and the local features and global features of the video frame image are extracted layer by layer through the convolutional layers and pooling layers in the VGG network to obtain the static feature representation of each video frame;

[0063] The video is input into a pre-trained I3D network, and the dynamic changes of the video are modeled in the time domain through the time dimension convolution operation of the convolutional layers in the I3D network to obtain the dynamic feature representation of the video;

[0064] S1-2, the static feature representation of the video frame and the dynamic feature representation of the video are fused by using a cross-Transformer network to obtain the video feature sequence.

[0065] Step S2 is specifically as follows:

[0066] S2-1, the user question is preprocessed, that is, the natural language processing operation of word segmentation, part-of-speech tagging and syntax analysis is performed on the user question to obtain a preprocessed question;

[0067] S2-2, the preprocessed question is converted into a question vector representation with semantic information by a BERT network.

[0068] Step S3 is specifically as follows:

[0069] S3-1, let the video feature sequence X = {X1, X2, …, X n}, wherein X i , the video feature sequence X is input into the cross-Transformer network, and the attention mechanism is used to obtain the visual features related to the user question;

[0070] S3-2, the visual features and the question vector representation are input into the cross-Transformer network to obtain the question guided video features X iq ;

[0071] S3-3, video feature X guided by the question i q Input the Bayesian differentiable neural network, train and optimize it through the back propagation algorithm, and obtain P(X i |X i-1 ,X i-2 ,…,X1), that is, the probability of the occurrence of the i-th frame video feature under the condition of the given first to i-1 frame video features, as shown in Figure 2 ;

[0072] The specific calculation formula of causal reasoning is as follows:

[0073]

[0074] Where ε is a random variable of Gaussian sampling distribution k I is the covariance matrix with k sampling numbers, and is the parameterized representation of the encoding network which can be learned, is the scientific parameterized representation of the decoding network, x u is the X i feature processed by the cross Transformer network, z u is the video feature output mapped by the learnable parameters.

[0075] Step S5, specifically as follows:

[0076] S5-1, input the question representation guided by the knowledge into the Transformer decoding network, and use the self-attention mechanism and the feedforward neural network layer of the Transformer decoding network to decode to obtain the generated question answer;

[0077] S5-2, compare and optimize the generated question answer with the standard question answer by using the cross entropy loss function, and obtain a more accurate question answer by minimizing the cross entropy loss function;

[0078] S5-3, input the video feature representation into the GPT-3 model, and combine the generated answer as a supervision signal to supervise the training of the Transformer network.

[0079] Step S6, specifically as follows:

[0080] S6-1, use the self-attention mechanism to calculate the attention weight when generating the answer;

[0081] S6-2 uses the charting library Matplotlib, the data visualization tool Plotly, and the visualization interaction tool D3.js to input the attention weights, the video feature sequences, and the question vector representations to generate charts and images, helping users intuitively understand the basis and reasoning process for answer generation;

[0082] S6-3, using explanatory descriptions, the attention weights are input to generate an analysis of the problem, supporting evidence for the generation of the answer, and an explanation of the reasoning path.

[0083] Accordingly, the present invention also provides a video question-answering system with interpretability and knowledge-heuristic capabilities, such as... Figure 3 As shown, it includes:

[0084] Video coding unit 1 is used to input video into the VGG network and the I3D network to extract video feature sequences;

[0085] Specifically, the video feature sequence includes static feature representations of video frames and dynamic feature representations of the video, and the extraction method is as follows:

[0086] The video is input into the pre-trained VGG network frame by frame. Through the convolutional and pooling layers in the VGG network, the local and global features of the video frame images are extracted layer by layer to obtain the static feature representation of each video frame.

[0087] The video is input into the pre-trained I3D network. Through the temporal convolution operation of the convolutional layer in the I3D network, the dynamic changes of the video are modeled in the temporal domain to obtain the dynamic feature representation of the video.

[0088] A cross-transformer network is used to fuse the static feature representations of the video frames and the dynamic feature representations of the video to obtain a video feature sequence.

[0089] Text encoding unit 2 is used to input user questions into the BERT network to extract question vector representations;

[0090] Specifically, the user question is preprocessed, which involves natural language processing operations such as word segmentation, part-of-speech tagging, and syntactic analysis to obtain the preprocessed question.

[0091] The preprocessed question is transformed into a question vector representation with semantic information using the BERT network.

[0092] The causal scene modeling unit 3 is used to input the video feature sequence and the problem vector representation into a cross-transformer network and a Bayesian differentiable neural network to obtain a causal representation of the video scene;

[0093] Specifically, let the video feature sequence be X = {X1, X2, ..., X...} n}, where X is the video feature of the i-th frame of the video. i The video feature sequence X is input into a cross-transformer network, and the attention mechanism is used to obtain visual features related to the user's question.

[0094] The visual features and the question vector representation are input into a Transformer network to obtain the question-guided video features X. i q ;

[0095] Video features that guide the question X i q Inputting a Bayesian differentiable neural network, and training and optimizing it using the backpropagation algorithm, yields P(X). i |X i-1 ,X i-2 ,…,X1), which is the probability of the video feature in frame i occurring given the video features from frame 1 to frame i-1;

[0096] The specific calculation formula for causal reasoning is as follows:

[0097]

[0098] Where ε is a Gaussian sampling distribution random variable, I k It is the covariance matrix with k samples. and It is a learnable parameterized representation of the encoding network. It is a parameterized representation of the decoding network science department, x u It is X i Features z after processing by the cross-Transformer network u It is the output of video features mapped by learnable parameters.

[0099] The knowledge introduction unit 4 is used to input the question vector representation and the causal representation of the video scene into the GPT-3 model, and generate a knowledge-guided question representation through the encoder-decoder structure of the model;

[0100] Answer generation unit 5 is used to input the knowledge-guided question representation into the Transformer decoding network to obtain the answer to the user question;

[0101] Specifically, the knowledge-guided question representation is input into the Transformer decoding network, and the self-attention mechanism and feedforward neural network layer of the Transformer decoding network are used to decode and obtain the generated question answer;

[0102] The generated answer to the question is compared with the standard answer to the question by using a cross-entropy loss function, and a more accurate answer to the question is obtained by minimizing the cross-entropy loss function;

[0103] The video feature representation is input into a GPT-3 model, and the generated answer is combined as a supervision signal to supervise the training of the Transformer network.

[0104] A visualization unit 6 is used to visualize the knowledge-guided question representation and the answer to the user question as an answer choice and a reasonable explanation generated by the user by using a self-attention mechanism, a visualization tool, and an explanatory description;

[0105] Specifically, the self-attention mechanism is used to calculate the attention weight when generating the answer;

[0106] The attention weight, the video feature sequence, and the question vector representation are input by using a chart drawing library Matplotlib, a data visualization tool Plotly, and a visualization interaction tool D3.js to generate charts and images, which assist the user in intuitively understanding the basis and reasoning process of answer generation;

[0107] The attention weight is input by using an explanatory description to generate an analysis of the question, supporting evidence for answer generation, and an explanation of the reasoning path.

[0108] Therefore, the present application can better understand the problem and the causal relationship between the video content and each detail in the video by introducing the scene modeling of causal reasoning, the large language model, and the knowledge-aided method, and reduce the dependence of the video question and answer model on external knowledge, thereby improving the accuracy of understanding and generating answers to difficult questions. Unlike other prior art, the present application also emphasizes the explanation and presentation of the output results and their reasoning process, which enhances the user's trust in the model output results, facilitates the user's supervision and improvement of the model, and significantly improves the user experience.

[0109] The above describes in detail the video question and answer method and system provided by the embodiment of the present application with explainability and knowledge-aided capability, and the principle and implementation mode of the present application are described by applying specific examples; the above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A video question-answering method with explainability and knowledge elicitation capability, characterized in that, The method comprises: inputting a video into a VGG network and an I3D network to extract a video feature sequence; inputting a user question into a BERT network to extract a question vector representation; inputting the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of a video scene; inputting the question vector representation and the causal representation of the video scene into a GPT-3 model to generate a knowledge-guided question representation through an encoding-decoding structure of the model; inputting the knowledge-guided question representation into a Transformer decoding network to obtain an answer to the user question; visualizing the knowledge-guided question representation and the answer to the user question into an answer selection and a reasonable explanation for generation of the answer that is easy for a user to understand through a self-attention mechanism, a visualization tool and an explanatory description; Specifically, the inputting the video feature sequence and the question vector representation into the cross-Transformer network and the Bayesian differentiable neural network to obtain the causal representation of the video scene comprises: A video feature sequence X={X1, X2, …, X n}, wherein the video feature of the i-th frame of the video is X i The video feature sequence X is input into the cross-Transformer network, and the attention mechanism is used to obtain the visual feature related to the user question. inputting the visual features into a cross Transformer network with the question vector representation to obtain question-guided video features X i q ; Video feature X guided by question i q Input Bayesian differentiable neural network, train and optimize by back propagation algorithm, get P(X i |X i-1 ,X i-2 …,X1), that is, the probability of the occurrence of the i-th frame video feature given the first to the i-1-th frame video feature The specific calculation formula of the causal reasoning is as follows: where ε is a random variable with Gaussian sampling distribution I k is the covariance matrix of k samples, and is the parameterized representation of the encoding network that can be learned, is the parameterized representation of the decoding network that can be learned, x u is X i the feature processed by the cross Transformer network, z u is the video feature output mapped by the learnable parameters.

2. The video question answering method with explainability and knowledge elicitation capability of claim 1, wherein, The inputting the video into the VGG network and the I3D network to extract the video feature sequence comprises: The video feature sequence comprises a static feature representation of a video frame and a dynamic feature representation of the video, and the extraction method is as follows: inputting the video into a pre-trained VGG network in units of frames, and extracting local features and global features of the video frame image layer by layer through convolutional layers and pooling layers in the VGG network to obtain the static feature representation of each video frame; inputting the video into a pre-trained I3D network, and modeling the dynamic changes of the video in the time domain through time dimension convolution operations of the convolutional layers in the I3D network to obtain the dynamic feature representation of the video; fusing the static feature representation of the video frame and the dynamic feature representation of the video by using a cross-Transformer network to obtain the video feature sequence.

3. The video question answering method with explainability and knowledge elicitation capability of claim 1, wherein, The inputting the user question into the BERT network to extract the question vector representation comprises: preprocessing the user question, i.e., performing natural language processing operations such as word segmentation, part-of-speech tagging and syntax analysis on the user question to obtain a preprocessed question; converting the preprocessed question into a question vector representation with semantic information through the BERT network.

4. The video question answering method with explainability and knowledge elicitation capability of claim 1, wherein, The inputting the knowledge-guided question representation into the Transformer decoding network to obtain the answer to the user question comprises: inputting the knowledge-guided question representation into the Transformer decoding network to decode the generated question answer by using a self-attention mechanism and a feedforward neural network layer of the Transformer decoding network; comparing and optimizing the generated question answer with a standard question answer by using a cross-entropy loss function, and obtaining a more accurate question answer by minimizing the cross-entropy loss function; inputting the video feature representation into the GPT-3 model, and combining the more accurate question answer as a supervision signal to supervise the training of the Transformer network.

5. The video question answering method with explainability and knowledge elicitation capability of claim 1, wherein, The knowledge guided question representation and the answer to the user question are visualized as an answer selection and a reasonable explanation of the generation by using the self-attention mechanism, the visualization tool and the explanatory description, specifically: The attention weight in the answer generation by using the self-attention mechanism; The attention weight, the video feature sequence and the question vector representation are inputted by using the chart drawing library Matplotlib, the data visualization tool Plotly and the visualization interactive tool D3.js to generate charts and images, which assist the user in intuitively understanding the basis and reasoning process of the answer generation; The attention weight is inputted by using the explanatory description to generate the analysis of the question, the supporting evidence of the answer generation and the explanation of the reasoning path.

6. A video question-answering system with explainability and knowledge elicitation capability, characterized in that, The system comprises: A video encoding unit configured to input a video into a VGG network and an I3D network to extract a video feature sequence; A text encoding unit configured to input a user question into a BERT network to extract a question vector representation; A causal scenario modeling unit configured to input the video feature sequence and the question vector representation into a cross-Transformer network and a Bayesian differentiable neural network to obtain a causal representation of a video scenario; A knowledge introduction unit configured to input the question vector representation and the causal representation of the video scenario into a GPT-3 model to generate a knowledge guided question representation through an encoding-decoding structure of the model; An answer generation unit configured to input the knowledge guided question representation into a Transformer decoding network to obtain an answer to the user question; A visualization unit configured to visualize the knowledge guided question representation and the answer to the user question as an answer selection and a reasonable explanation of the generation by using the self-attention mechanism, the visualization tool and the explanatory description; Specifically, the causal scenario modeling unit is configured to input the video feature sequence and the question vector representation into the cross-Transformer network and the Bayesian differentiable neural network to obtain the causal representation of the video scenario, specifically: A video feature sequence X={X1, X2, …, X n} is recorded, wherein the video feature of the i-th frame of the video is X i The video feature sequence X is input into the cross-Transformer network, and the attention mechanism is used to obtain the visual feature related to the user question. inputting the visual features into a cross Transformer network with the question vector representation to obtain question-guided video features X i q ; Video feature X guided by question i q Input Bayesian differentiable neural network, train and optimize by back propagation algorithm, get P(X i |X i-1 ,X i-2 …,X1), that is, the probability of the occurrence of the i-th frame video feature given the first to the i-1-th frame video feature The specific calculation formula of the causal reasoning is as follows: where ε is a random variable with Gaussian sampling distribution I k is a covariance matrix with k samples, and is a parameterized representation of the encoding network that can be learned, is a parameterized representation of the decoding network that can be learned, x u is X i is the feature after being processed by the cross Transformer network, z u is the video feature output after being mapped by the learnable parameters.

7. The video question answering system with explainability and knowledge elicitation capability of claim 6, wherein, The video encoding unit is configured to input a video into a VGG network and an I3D network to extract a video feature sequence, specifically: The video feature sequence comprises a static feature representation of a video frame and a dynamic feature representation of a video, and the extraction method is as follows: The video is inputted into a pre-trained VGG network in units of frames, and the local features and the global features of the video frame image are extracted layer by layer through the convolutional layers and the pooling layers in the VGG network to obtain the static feature representation of each video frame; The video is inputted into a pre-trained I3D network, and the dynamic changes of the video are modeled in the time domain through the time dimension convolution operation of the convolutional layers in the I3D network to obtain the dynamic feature representation of the video; The static feature representation of the video frame and the dynamic feature representation of the video are fused by using the cross-Transformer network to obtain the video feature sequence.

8. The video question answering system with explainability and knowledge elicitation capability of claim 6, wherein, The text encoding unit is configured to input a user question into a BERT network to extract a question vector representation, specifically: Preprocessing the user question, that is, performing natural language processing operations such as word segmentation, part-of-speech tagging, and syntax analysis on the user question to obtain a preprocessed question; Converting the preprocessed question into a question vector representation with semantic information through a BERT network.

9. The video question answering system with explainability and knowledge elicitation capability of claim 6, wherein, The answer generation unit is configured to input the knowledge-guided question representation into a Transformer decoding network to obtain an answer to the user question, specifically: Inputting the knowledge-guided question representation into a Transformer decoding network, and decoding the generated question answer using the self-attention mechanism and the feedforward neural network layer of the Transformer decoding network; Comparing and optimizing the generated question answer with a standard question answer using a cross-entropy loss function, and obtaining a more accurate question answer by minimizing the cross-entropy loss function; Inputting the video feature representation into a GPT-3 model, and combining the more accurate question answer as a supervision signal to supervise the training of the Transformer network.

10. The video question answering system with explainability and knowledge elicitation capability of claim 6, wherein, The visualization unit is configured to visualize the knowledge-guided question representation and the answer to the user question as a reasonable explanation for the selection and generation of the answer that is easy for the user to understand through a self-attention mechanism, visualization tools, and explanatory descriptions, specifically: Using a self-attention mechanism to calculate the attention weight when generating an answer; Using a charting library Matplotlib, a data visualization tool Plotly, and a visualization interaction tool D3.js to input the attention weight, the video feature sequence, and the question vector representation to generate charts and images, which assist the user in intuitively understanding the basis and reasoning process of answer generation; Using explanatory descriptions to input the attention weight to generate an analysis of the question, supporting evidence for answer generation, and an explanation of the reasoning path.

Images