Answer generation method for video question answering system

Abstract

The application relates to the technical field of video question answering systems, and provides a video question answering system answer generation method, which comprises the following steps: independently modeling a plurality of acquired modal features respectively to generate corresponding situational representations of the modal features; determining final layer node values of a graph structure corresponding to each modal feature based on the corresponding situational representations of the modal features; performing target round reasoning based on the final layer node values of the graph structure corresponding to each modal feature to generate a final reasoning result; and generating a target answer based on the final reasoning result and the corresponding situational representations of the modal features. The video question answering system answer generation method can jointly process features extracted from an abstract modal, a visual modal and a question modal, and improves the accuracy and relevance of the target answer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video question answering system, and particularly relates to a video question answering system answer generation method. BACKGROUND

[0002] The main task of a video question answering system is to give a video and a question related to the content in the video, analyze the video content and understand the question description, and give the correct answer to the question. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a video question answering system answer generation method to improve the accuracy of the generated answer.

[0004] The present application also proposes a video question answering system answer generation device.

[0005] The present application also proposes an electronic device.

[0006] The present application also proposes a non-transitory computer readable storage medium.

[0007] The present application also proposes a computer program product.

[0008] According to the video question answering system answer generation method of the first aspect of the present application, comprising:

[0009] Respectively model the obtained multiple modal features to generate contextualized representations corresponding to each modal feature;

[0010] Based on the contextualized representations corresponding to each modal feature, determine the final layer node values of the graph structure corresponding to each modal feature;

[0011] Based on the final layer node values of the graph structure corresponding to each modal feature, perform target round reasoning to generate a final reasoning result;

[0012] Based on the final reasoning result and the contextualized representations corresponding to each modal feature, generate a target answer;

[0013] Among them, the multiple modal features include a summary modal feature corresponding to a to-be-answered video, a video modal feature corresponding to the to-be-answered video, and a question modal feature corresponding to a to-be-answered question.

[0014] According to the answer generation method of the video question answering system provided in the embodiments of the present application, the inference method based on the graph neural network and the cross-modal attention mechanism can jointly process the features extracted from the summary modal, the visual modal and the question modal, and is used for feature extraction of the video question answering system, so as to effectively extract event information and event correlation information in the video to be answered, and improve the accuracy and precision of feature extraction. By using the adaptive multi-modal fusion method based on the question guidance, the content with high relevance to the question to be answered and the video to be answered can be extracted to generate the target answer, so as to improve the accuracy and relevance of the target answer.

[0015] According to an embodiment of the present application, the final layer node value of the graph structure corresponding to each modal feature is determined based on the contextualized representation corresponding to the modal feature.

[0016] Based on the contextualized representation corresponding to each modal feature, a graph structure corresponding to each modal feature is established.

[0017] Based on the graph structure, inference of the cross-modal attention mechanism is performed to generate the final layer node value of the graph structure corresponding to each modal feature.

[0018] According to an embodiment of the present application, the final layer node value of the graph structure corresponding to each modal feature is determined based on the contextualized representation corresponding to the modal feature.

[0019] Based on the semantic correlation value between each node in the graph structure, the initial node value of the graph structure is updated using a graph convolutional neural network to generate a first node value.

[0020] The target modal feature in the plurality of modal features is taken as a query modal to determine the cross-modal attention of the target modal feature and other modal features.

[0021] Based on the cross-modal attention, the first node value of the graph structure corresponding to the target modal feature is updated to generate the final layer node value of the graph structure corresponding to the target modal feature.

[0022] According to an embodiment of the present application, the number of nodes of the graph structure is the length of the contextualized representation corresponding to the modal feature corresponding to the graph structure.

[0023] According to an embodiment of the present application, the target round inference is performed based on the final layer node value of the graph structure corresponding to each modal feature to generate a final inference result, which comprises:

[0024] Based on the final layer node value of the graph structure corresponding to each modal, the question-guided modal feature under the current round of inference is determined.

[0025] merge the question-guided modal features corresponding to each modality under the current round reasoning to determine a first reasoning result under the current round reasoning;

[0026] determine that the current round is less than the target round, and generate a first reasoning result under a next round reasoning based on the first reasoning result under the current round reasoning;

[0027] determine that the current round is equal to the target round, and determine the first reasoning result under the current round reasoning as the final reasoning result.

[0028] According to an embodiment of the present application, the generating of the target answer based on the final reasoning result and the contextualized representation corresponding to each modality feature comprises:

[0029] merging the final reasoning result, the contextualized representation corresponding to the summary modality feature, and the contextualized representation corresponding to the video modality feature to generate a final representation;

[0030] generating the target answer based on the question type of the question to be answered and the final representation.

[0031] According to an embodiment of the present application, the generating of the target answer based on the question type of the question to be answered and the final representation comprises:

[0032] determining that the question type is a selection type question, and performing classification task processing on the final representation to generate the target answer;

[0033] determining that the question type is a numerical type question, and performing regression task processing on the final representation to generate the target answer.

[0034] According to an embodiment of the present application, the independent modeling of the obtained multiple modality features respectively to generate the contextualized representation corresponding to each modality feature comprises:

[0035] independent modeling of each of the modality features based on a recurrent neural network to generate the contextualized representation corresponding to each modality feature.

[0036] According to an embodiment of the present application, before the independent modeling of the obtained multiple modality features respectively to generate the contextualized representation corresponding to each modality feature, the method comprises:

[0037] generating the summary modality feature based on the shape feature and the optical flow feature extracted from the to-be-answered video;

[0038] generating the video modality feature based on the shape feature and the motion feature extracted from the to-be-answered video;

[0039] generate the question modality feature based on word vectors extracted from the question to be answered.

[0040] According to an embodiment of the present application, the generating the summary modality feature based on the appearance feature and the optical flow feature extracted from the video to be asked and answered comprises:

[0041] extracting the appearance feature and the optical flow feature from the video to be asked and answered by using a 2D convolutional neural network;

[0042] generating a dense video summary based on the appearance feature and the optical flow feature by using a dense video summary generation method;

[0043] generating a plurality of summary representation vectors based on the dense video summary by using a word vector embedding method and a recurrent neural network;

[0044] connecting the plurality of summary representation vectors to generate the summary modality feature.

[0045] According to an embodiment of the present application, the generating the question modality feature based on word vectors extracted from the question to be answered comprises:

[0046] generating an embedding vector for each word in the question to be answered by using a word vector embedding method;

[0047] connecting the embedding vector corresponding to each word to generate the question modality feature.

[0048] The answer generation device of the video question-answering system according to the second aspect embodiment of the present application comprises:

[0049] a first processing module configured to independently model a plurality of modality features obtained respectively to generate contextualized representations corresponding to the modality features;

[0050] a second processing module configured to determine final layer node values of a graph structure corresponding to the modality features based on the contextualized representations corresponding to the modality features;

[0051] a third processing module configured to perform target round reasoning based on the final layer node values of the graph structure corresponding to the modality features to generate a final reasoning result;

[0052] a fourth processing module configured to generate a target answer based on the final reasoning result and the contextualized representations corresponding to the modality features.

[0053] The plurality of modality features comprise a summary modality feature corresponding to a video to be asked and answered, a video modality feature corresponding to the video to be asked and answered, and a question modality feature corresponding to a question to be answered.

[0054] According to the answer generation device of the video question answering system in the embodiment of the present application, the inference method based on the graph neural network and the cross-modal attention mechanism can jointly process the features extracted from the summary modal, the visual modal and the question modal, and is used for feature extraction of the video question answering system, so as to effectively extract event information and event correlation information in the video to be answered, and improve the accuracy and precision of feature extraction; by using the adaptive multi-modal fusion method based on question guidance, the content with high relevance to the question to be answered and the video to be answered can be extracted to generate the target answer, so as to improve the accuracy and relevance of the target answer.

[0055] According to the electronic device in the third aspect of the embodiment of the present application, the memory, the processor and the computer program stored in the memory and executable on the processor are included, and the processor implements the answer generation method of the video question answering system in any of the above aspects when executing the computer program.

[0056] According to the non-transitory computer readable storage medium in the fourth aspect of the embodiment of the present application, the computer program is stored thereon, and the computer program is executed by the processor to implement the answer generation method of the video question answering system in any of the above aspects.

[0057] According to the computer program product in the fifth aspect of the embodiment of the present application, the computer program is included, and the computer program is executed by the processor to implement the answer generation method of the video question answering system in any of the above aspects.

[0058] The one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0059] According to the answer generation device of the video question answering system in the embodiment of the present application, the inference method based on the graph neural network and the cross-modal attention mechanism can jointly process the features extracted from the summary modal, the visual modal and the question modal, and is used for feature extraction of the video question answering system, so as to effectively extract event information and event correlation information in the video to be answered, and improve the accuracy and precision of feature extraction; by using the adaptive multi-modal fusion method based on question guidance, the content with high relevance to the question to be answered and the video to be answered can be extracted to generate the target answer, so as to improve the accuracy and relevance of the target answer.

[0060] Further, by performing dense video summary on the content of the video to be answered to extract event information and event correlation information in the video to be answered, the summary modal features are generated for data processing of the video question answering system, which can effectively deal with complex events with concurrency, sequence or causality in the video.

[0061] Further, different processing manners are adopted to generate target answers based on different question types, which can enable the user to select the best calculation manner based on actual conditions, thereby helping to further improve the accuracy of the generated target answers.

[0062] Additional aspects and advantages of the application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0064] Figure 1 is one of the flowcharts of the answer generation method of the video question and answer system provided by the embodiments of the present application;

[0065] Figure 2 is the second flowchart of the answer generation method of the video question and answer system provided by the embodiments of the present application;

[0066] Figure 3 is the third flowchart of the answer generation method of the video question and answer system provided by the embodiments of the present application;

[0067] Figure 4 is the fourth flowchart of the answer generation method of the video question and answer system provided by the embodiments of the present application;

[0068] Figure 5 is the structural schematic diagram of the answer generation device of the video question and answer system provided by the embodiments of the present application;

[0069] Figure 6 is the structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application will be further described in detail below in combination with the drawings and embodiments. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0071] In the description of the embodiments of the present application, it should be noted that the terms "first", "second", "third" are only used for description purpose, and cannot be understood as indicating or implying relative importance.

[0072] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0073] The video question answering system mainly includes the following answer methods:

[0074] 1) Extract one or several key frames in the video, and then use the existing image question answering algorithm to obtain the answer by extracting the features of a single picture and extracting or embedding the text of the question;

[0075] 2) On the basis of 1), the appearance feature and / or motion feature in the video are extracted, and the answer is obtained by combining the text extraction or text embedding learning of the question;

[0076] 3) On the basis of 2), a temporal attention mechanism on the video is introduced to focus on some features extracted in the time dimension to obtain the answer; 4) On the basis of 2), a memory mechanism of a neural network is introduced to deepen the memory and use of video related information features to obtain the answer.

[0077] The inventor found that the above methods are difficult to effectively deal with complex events with concurrent, sequential or causal relationships in the video, and it is more difficult to accurately understand the question or video content for complex questions and complex implicit features, resulting in low accuracy of the obtained answers.

[0078] The video question answering system of the embodiments of the present application will be described below. Figures 1-4 The answer generation method of the video question answering system of the embodiments of the present application is described.

[0079] It should be noted that the execution subject of the answer generation method of the video question and answer system can be an answer generation device of the video question and answer system, or can be a server in communication connection with the video question and answer system, or can be a user terminal in communication connection with the video question and answer system, including but not limited to a mobile phone, a watch, a smart home appliance, or a robot, etc.

[0080] The answer generation method of the video question and answer system is suitable for deployment of various products with visual perception, such as smart phones, service robots, and smart appliances, etc., and various information retrieval and analysis platforms or programs, such as short videos, science and education, entertainment, and news, etc.

[0081] As shown in Figure 1 The answer generation method of the video question and answer system includes steps 110, 120, 130, and 140.

[0082] In step 110, the obtained multiple modal features are independently modeled respectively to generate contextualized representations corresponding to the modal features.

[0083] In this step, the contextualized representation is an expression of each modal feature in the current context, which is used to represent the meaning in the current context.

[0084] The multiple modal features include: question modal features F q corresponding to the question to be answered Q, summary modal features F c corresponding to the video to be answered Q, and video modal features F v .

[0085] The video to be answered V includes multiple video frames, and the video frames include background information and object information.

[0086] The question to be answered Q is a question designed based on the video content of the video to be answered; the question to be answered Q can be in the form of text.

[0087] The summary modal features F c are generated by using a dense video captioning method on the video to be answered V. The dense video captioning method is used to automatically generate descriptive sentences based on the visual information of the video, so as to extract event information and event relevance information in the video to be answered, and save such information into the summary modal for data processing of the video question and answer system.

[0088] In some embodiments, step 110 can include independently modeling each modal feature based on a recurrent neural network to generate a contextualized representation corresponding to the modal feature.

[0089] In this embodiment, the summary modal feature F c , the video modal feature F v , and the question modal feature F q are independently modeled using a recurrent neural network, and the hidden variable outputs of each step of the recurrent neural network are taken as the generated contextualized representations.

[0090] The recurrent neural network can be a GRU, etc.

[0091] Specifically, the summary modal feature F c corresponds to the contextualized representation record C l , the length of C l is Nc; the video modal feature F v corresponds to the contextualized representation record V l , the length of V l is Nv; and the question modal feature F q corresponds to the contextualized representation record Q l , the length of Q l is Nq.

[0092] According to the answer generation method of the video question answering system provided in the embodiments of the present application, the recurrent neural network is used to independently model each modal feature, and generate the contextualized representation corresponding to each modal feature, which has high accuracy and precision.

[0093] As shown in FIG. 1, in some embodiments, before step 110, the method can further include: Figure 4

[0094] Based on the optical flow feature and the shape feature extracted from the to-be-answered video, the summary modal feature is generated;

[0095] Based on the motion feature and the shape feature extracted from the to-be-answered video, the video modal feature is generated;

[0096] Based on the word vector extracted from the to-be-answered question, the question modal feature is generated.

[0097] In this embodiment, the shape feature includes the contour / region feature of the target object in the video frame.

[0098] The optical flow feature is the instantaneous velocity feature of the pixel motion of the spatial motion object on the video frame.

[0099] The motion feature is the motion feature of the target object in the video frame.

[0100] In an embodiment, based on the optical flow feature and the shape feature extracted from the to-be-answered video, the summary modal feature can include:

[0101] ​The 2D convolutional neural network is used to extract the appearance feature and the optical flow feature from the to-be-answered video.

[0102] Based on the appearance feature and the optical flow feature, a dense video summary generation method is used to generate a dense video summary.

[0103] Based on the dense video summary, a word vector embedding method and a recurrent neural network are used to generate a plurality of summary representation vectors.

[0104] The plurality of summary representation vectors are connected to generate a summary modal feature.

[0105] In this embodiment, the 2D convolutional neural network can be used to generate the appearance feature F a and the optical flow feature from the to-be-answered video V, and further use a dense video summary generation method (such as Masked Transformer) to generate a dense video summary.

[0106] The 2D convolutional neural network can be ResNet and BN-Inception.

[0107] For each generated dense video summary, a word vector embedding method and a recurrent neural network are used to generate a summary representation vector, and all summary representation vectors are connected to obtain a summary modal feature F c .

[0108] The length of the summary modal feature F c is Nc.

[0109] The recurrent neural network can be GloVe and GRU.

[0110] The 2D convolutional neural network is used to extract the appearance feature F a from the to-be-answered video V.

[0111] The 2D convolutional neural network can be ResNet or VGG.

[0112] The 3D convolutional neural network (such as C3D) is used to generate the motion feature F m from the to-be-answered video V; and a fully connected layer network is used to generate the video modal feature F v from F a and F m .

[0113] The length of the video modal feature F v is Nv.

[0114] In some embodiments, based on the word vector extracted from the to-be-answered question, the question modal feature can include:

[0115] An embedding vector of each word in the question to be answered is generated by using a word embedding method.

[0116] The embedding vectors corresponding to each word are connected to generate the question modality feature.

[0117] In this embodiment, an embedding vector (i.e., a word vector) of each word in the question to be answered Q can be generated by using a word embedding method (such as GloVe), and the embedding vectors of each word are connected into a sequence to generate the question modality feature F q .

[0118] The length of the question modality feature F q is denoted as Nq.

[0119] According to the answer generation method of the video question-answering system provided in the embodiments of the present application, the event information and event relevance information in the video to be answered are extracted by performing dense video summarization on the video content to be answered, so as to generate a summary modality feature for data processing of the video question-answering system, which can effectively deal with complex events with concurrent, sequential or causal relationships in the video.

[0120] In step 120, the final layer node value of the graph structure corresponding to each modality feature is determined based on the contextualized representation corresponding to each modality feature.

[0121] In this step, the graph structure includes a plurality of nodes, and each pair of nodes has semantic relevance.

[0122] The final layer node value is the final node feature value obtained after the graph structure is updated multiple times.

[0123] The implementation of this step will be described below through specific embodiments.

[0124] In some embodiments, step 120 can include:

[0125] Based on the contextualized representation corresponding to each modality feature, a graph structure corresponding to each modality feature is established respectively;

[0126] Based on the graph structure, the inference of the cross-modal attention mechanism is performed to generate the final layer node value of the graph structure corresponding to each modality feature.

[0127] In this embodiment, it should be noted that the number of nodes of the graph structure is the length of the contextualized representation corresponding to the modality feature corresponding to the graph structure.

[0128] For example, in the actual execution process, the embedding vectors of the words in the question to be answered Q can be generated based on the C l , V l and Q lrespectively, wherein the number of nodes of each graph structure is the length of the contextualized representation of the modality feature corresponding to the graph structure, that is, the number of nodes of the graph structure corresponding to the summary modality feature is Nc, the number of nodes of the graph structure corresponding to the video modality feature is Nv, and the number of nodes of the graph structure corresponding to the question modality feature is Nq.

[0129] After obtaining the graph structure corresponding to each modality feature, graph reasoning and reasoning based on the cross-modality attention mechanism can be performed to obtain the final layer node value of the graph structure corresponding to each modality feature.

[0130] According to the answer generation method of the video question answering system provided in the embodiments of the present application, by constructing the graph structure of each modality feature respectively and performing reasoning based on the cross-modality attention mechanism to generate the final layer node value, the relationship between each node can be accurately established, and the accuracy and precision are high.

[0131] As shown in FIG. 1, Figure 2 in some embodiments, the reasoning based on the cross-modality attention mechanism based on the graph structure to generate the final layer node value of the graph structure corresponding to each modality feature can include:

[0132] updating the initial node value of the graph structure by using a graph convolutional neural network based on the pairwise semantic correlation value of all nodes in the graph structure to generate the first node value;

[0133] taking the target modality feature in the plurality of modality features as a query modality to determine the cross-modality attention of the target modality feature and other modality features;

[0134] updating the first node value of the graph structure corresponding to the target modality feature based on the cross-modality attention to generate the final layer node value of the graph structure corresponding to the target modality feature.

[0135] In this embodiment, for the constructed graph structure, the corresponding node value is the initial node value.

[0136] For each graph structure, the pairwise semantic correlation value of all nodes in the graph structure is calculated to obtain the pairwise semantic correlation value G l C of the summary modality feature, the pairwise semantic correlation value G l V of the video modality feature, and the pairwise semantic correlation value G l Q of the question modality feature, respectively.

[0137] Then, the initial node value is updated by using a graph convolutional neural network to obtain the first node value of the summary modality feature The first node value corresponding to the video modality feature and the first node value corresponding to the question modality feature

[0138] The graph convolutional neural network can be GCN, etc.

[0139] Taking the video modality feature as an example, as shown in Figure 2 , specifically, the formula can be used:

[0140]

[0141] The video modality feature F v corresponding to the two-by-two semantic correlation value G l V ; wherein V l is the contextualized representation corresponding to the video modality feature F v .

[0142] The first node value corresponding to the video modality feature is determined by the formula:

[0143]

[0144] ; wherein, is the two-by-two semantic correlation value corresponding to the video modality feature F v , V l is the contextualized representation corresponding to the video modality feature F v , and W is a training weight, wherein W is in the form of a matrix.

[0145] For the other two modality features, their corresponding calculation methods are similar to the above formula, which will not be repeated here.

[0146] After obtaining the two-by-two semantic correlation value G l C corresponding to the summary modality feature, the two-by-two semantic correlation value G l V corresponding to the video modality feature, and the two-by-two semantic correlation value G l Q corresponding to the question modality feature, inference based on the cross-modality attention mechanism can be performed.

[0147] Specifically, continuing to refer to Figure 2 , for each of the three modalities of the summary modality feature, the video modality feature and the question modality feature, as the query modality Q (Query), the cross-modality attention with the other two modalities is calculated and used to update the node feature of the modality.

[0148] The feature of the modality selected in the attention mechanism is denoted as M Q ​, M K , M V , where K (Key) and V (Value) are the same modality, Q (Query) is another modality.

[0149] Then the cross-modal attention mechanism is calculated as follows:

[0150]

[0151] where, M Q is the feature of the Query modality, M K is the feature of the Key modality, M V is the feature of the Value modality, and d is the vector dimension.

[0152] With reference to Figure 2 , the following continues to take the video modality feature F v as an example of the Query modality M Q to describe this embodiment of the application.

[0153] First, the first node value corresponding to the summary modality feature is converted by the following formula: the first node value corresponding to the video modality feature is converted by the following formula: and the first node value corresponding to the question modality feature is converted by the following formula:

[0154]

[0155]

[0156]

[0157] where, and are the training weights of the modality features corresponding to each subscript, respectively; is the first node value corresponding to the video modality feature, is the first node value corresponding to the question modality feature, is the first node value corresponding to the summary modality feature.

[0158] After the conversion operation, the obtained and are used to calculate the cross-modal attention mechanism by the following formula:

[0159]

[0160]

[0161] where, V vql is the video modality feature F​v and the cross-modal attention of the question modal feature F q V vcl is the video modal feature F v and the cross-modal attention of the summary modal feature F c .

[0162] After obtaining the cross-modal attention, the new (next layer) node feature value can be calculated by the formula:

[0163]

[0164] , wherein V l+1 is the next layer node value of the graph structure corresponding to the video modal feature.

[0165] Figure 2 The flowchart from to V l+1 is shown.

[0166] Step 120 is repeated multiple times, and the final layer node feature values of the graph structures corresponding to the modal features are obtained, wherein the final layer node feature value of the graph structure corresponding to the summary modal feature is the final layer node feature value of the graph structure corresponding to the video modal feature is the final layer node feature value of the graph structure corresponding to the question modal feature is

[0167] According to the answer generation method of the video question answering system provided in the embodiments of the present application, each of the three modalities is taken as a query modality (query) to calculate the cross-modal attention with the other two modalities, and the cross-modal attention is used to update the node features of the modality, which can better extract video content evidence related to the question to be answered and related to the event, and is used to generate the final answer of the video question answering system, thereby improving the accuracy of the generated result.

[0168] Step 130, based on the final layer node values of the graph structures corresponding to the modal features, performing target round reasoning to generate a final reasoning result;

[0169] In this step, the target round is the total number of rounds of reasoning.

[0170] The value of the target round can be based on user customization, which is not limited by the present application.

[0171] In the actual execution process, the final layer node feature value of the graph structure corresponding to the summary modal feature can be the final layer node feature value of the graph structure corresponding to the video modal feature is the final layer node feature value of the graph structure corresponding to the question modal feature is The final reasoning result h can be obtained by using the modal fusion model to perform Nr rounds of reasoning Nr , and Nr is a positive integer.

[0172] The modal fusion model can be an HME model.

[0173] The specific implementation of this step will be described below. Figure 3

[0174] In some embodiments, step 130 can include:

[0175] Based on the final layer node value of the graph structure corresponding to each modality, the modal feature guided by the question corresponding to each modality under the current round of reasoning is determined.

[0176] The modal features guided by the questions corresponding to each modality under the current round of reasoning are merged to determine the first reasoning result under the current round of reasoning.

[0177] If the current round is less than the target round, the first reasoning result under the next round of reasoning is generated based on the first reasoning result under the current round of reasoning.

[0178] If the current round is equal to the target round, the first reasoning result under the current round of reasoning is determined as the final reasoning result.

[0179] Figure 3 The flowchart of the single reasoning process is shown.

[0180] The implementation of this embodiment will be described taking the tth round as an example.

[0181] During the tth round of reasoning, the question-guided modal features of each modality can be generated based on the following formula and

[0182]

[0183]

[0184]

[0185] wherein W is a training weight corresponding to the modal features of each subscript, which is in the form of a matrix; b is a first weight corresponding to the modal features of each superscript, which is in the form of a vector; q in the above formula is the last dimension of the question Q to be answered, and h t-1 is the reasoning result corresponding to the (t-1)th round.

[0186]

[0187] ​

[0188]

[0189] wherein, is the question-guided summary modal feature, is the question-guided video modal feature, is the question-guided question modal feature, is the final layer node feature value of the graph structure corresponding to the summary modal feature, is the final layer node feature value of the graph structure corresponding to the video modal feature, is the final layer node feature value of the graph structure corresponding to the question modal feature.

[0190] With reference to Figure 3 , the three modal features are then merged to generate the reasoning result h corresponding to the t-th round based on the following formula t .

[0191]

[0192]

[0193]

[0194] wherein, h t-1 is the reasoning result corresponding to the t-1-th round, is the question-guided summary modal feature, is the question-guided video modal feature, is the question-guided question modal feature, and W is a training weight and b is a first weight.

[0195] The reasoning process is repeated for Nr rounds of reasoning, and the final reasoning result h Nr is obtained.

[0196] According to the answer generation method of the video question answering system provided in the embodiments of the present application, through multiple rounds of reasoning, the relevance of the extracted features to the content of the to-be-answered question and the to-be-answered video is improved, thereby improving the accuracy and relevance of the finally generated target answer.

[0197] Step 140, generating a target answer based on the final reasoning result and the contextualized representation corresponding to each modal feature.

[0198] In this step, the target answer is an answer about the to-be-answered video related to the to-be-answered question, which is automatically generated by the system after inputting the to-be-answered video and the to-be-answered question.

[0199] The contextualized representation corresponding to each modal feature includes: the contextualized representation C c corresponding to the summary modal feature Fl ; video modality feature F v corresponding contextualized representation V l ; and question modality feature F q corresponding contextualized representation Q l .

[0200] The specific implementation of step 140 is described below.

[0201] As shown in some embodiments, step 140 can include: Figure 4

[0202] merging the video modality feature corresponding contextualized representation, the summary modality feature corresponding contextualized representation, and the final inference result to generate a final representation;

[0203] generating a target answer based on the final representation and the question type of the question to be answered.

[0204] In this embodiment, for example, as shown in Figure 4 , the final inference result h Nr may be merged with the video modality feature corresponding contextualized representation V l , the summary modality feature corresponding contextualized representation C l to obtain a final representation S a .

[0205] Then generate a target answer based on the final representation S a .

[0206] It should be noted that in this application, different generation methods can be further used for different question types.

[0207] Among them, the question type includes a selection type question and a number type question.

[0208] According to the answer generation method of the video question answering system provided in the embodiments of the present application, through adaptive multi-modal fusion based on question guidance, the video content evidence related to the question and the event can be better extracted, which is used to generate the final answer of the video question answering system, thereby improving the accuracy of the generated target answer.

[0209] In some embodiments, generating a target answer based on the question type of the question to be answered and the final representation can include:

[0210] determining that the question type is a selection type question, performing classification task processing on the final representation to generate a target answer;

[0211] determining that the question type is a number type question, performing regression task processing on the final representation to generate a target answer.

[0212] ​In this embodiment, for the selection type question, a classification task can be processed.

[0213] Specifically, the target answer can be generated by the following formula:

[0214]

[0215]

[0216] wherein, S is the target answer, is the final representation, a is the training weight, and b is the first weight. w is the first weight.

[0217] For the numerical type question, a regression task can be processed.

[0218]

[0219] wherein, S is the target answer, is the final representation, a is the training weight, and b is the first weight. n is the first weight.

[0220] In this embodiment, different processing methods are used to generate the target answer based on different question types, which can enable the user to select the best calculation method based on the actual situation, thereby helping to further improve the accuracy of the generated target answer.

[0221] According to the answer generation method of the video question answering system provided in the embodiments of the present application, by using the reasoning method based on the graph neural network and the cross-modal attention mechanism, the features extracted from the summary modal, the visual modal and the question modal can be jointly processed, which is used for feature extraction of the video question answering system, so as to effectively extract event information and event correlation information in the video to be questioned and answered, and improve the accuracy and precision of feature extraction; by using the adaptive multi-modal fusion method based on question guidance, the content with high relevance to the question to be answered and the video to be questioned and answered can be extracted to generate the target answer, thereby improving the accuracy and relevance of the target answer.

[0222] The answer generation device of the video question answering system provided in the embodiments of the present application is described below, and the answer generation device of the video question answering system described below can be correspondingly referred to the answer generation method of the video question answering system described above.

[0223] As shown in the figure, Figure 5 the answer generation device of the video question answering system comprises a first processing module 510, a second processing module 520, a third processing module 530 and a fourth processing module 540. ​​

[0224] The first processing module 510 is configured to independently model the obtained multiple modal features respectively, and generate situational representations corresponding to the modal features; the multiple modal features include a video modal feature corresponding to the to-be-answered video, a question modal feature corresponding to the to-be-answered question, and a summary modal feature corresponding to the to-be-answered video;

[0225] The second processing module 520 is configured to determine final layer node values of the graph structures corresponding to the modal features based on the situational representations corresponding to the modal features;

[0226] The third processing module 530 is configured to perform target round reasoning based on the final layer node values of the graph structures corresponding to the modal features, and generate a final reasoning result;

[0227] The fourth processing module 540 is configured to generate a target answer based on the final reasoning result and the situational representations corresponding to the modal features.

[0228] According to the answer generation apparatus of the video question-answering system provided in the embodiments of the present application, the reasoning method based on the graph neural network and the cross-modal attention mechanism can jointly process the features extracted from the summary modal, the visual modal, and the question modal, and is used for feature extraction of the video question-answering system, so as to effectively extract event information and event correlation information in the to-be-answered video, and improve the accuracy and precision of feature extraction. By using the adaptive multi-modal fusion method based on question guidance, content having high relevance to the to-be-answered question and the to-be-answered video can be extracted to generate a target answer, so as to improve the accuracy and relevance degree of the target answer.

[0229] In some embodiments, the second processing module 520 can be further configured to:

[0230] establish the graph structures corresponding to the modal features respectively based on the situational representations corresponding to the modal features;

[0231] perform reasoning of the cross-modal attention mechanism based on the graph structures, and generate the final layer node values of the graph structures corresponding to the modal features.

[0232] In some embodiments, the second processing module 520 can be further configured to:

[0233] update the initial node values of the graph structures by using a graph convolutional neural network based on the two-by-two semantic correlation values of all nodes in the graph structures, and generate first node values;

[0234] determine the cross-modal attention between the target modal feature and the other modal features by taking the target modal feature in the multiple modal features as a query modal;

[0235] The first node value of the graph structure corresponding to the target modality feature is updated based on cross-modal attention, and a final layer node value of the graph structure corresponding to the target modality feature is generated.

[0236] In some embodiments, the third processing module 530 can be further configured to:

[0237] Based on the final layer node value of the graph structure corresponding to each modality, the modality feature guided by the question corresponding to each modality under the current round of reasoning is determined.

[0238] The modality feature guided by the question corresponding to each modality under the current round of reasoning is merged to determine the first inference result under the current round of reasoning.

[0239] It is determined that the current round is less than the target round, and the first inference result under the next round of reasoning is generated based on the first inference result under the current round of reasoning.

[0240] It is determined that the current round is equal to the target round, and the first inference result under the current round of reasoning is determined as the final inference result.

[0241] In some embodiments, the fourth processing module 540 can be further configured to:

[0242] The contextualized representation corresponding to the summary modality feature, the contextualized representation corresponding to the video modality feature, and the final inference result are merged to generate a final representation.

[0243] Based on the question type of the question to be answered and the final representation, a target answer is generated.

[0244] In some embodiments, the fourth processing module 540 can be further configured to:

[0245] It is determined that the question type is a selection type question, and the final representation is processed by a classification task to generate a target answer.

[0246] It is determined that the question type is a numerical type question, and the final representation is processed by a regression task to generate a target answer.

[0247] In some embodiments, the first processing module 510 can be further configured to:

[0248] Based on the recurrent neural network, each modality feature is independently modeled to generate a contextualized representation corresponding to each modality feature.

[0249] In some embodiments, the apparatus can further include a fifth processing module configured to:

[0250] Before independently modeling the obtained multiple modality features to generate a contextualized representation corresponding to each modality feature, based on the optical flow feature and the shape feature extracted from the to-be-answered video, a summary modality feature is generated.

[0251] Based on the motion and shape features extracted from the video to be answered, video modal features are generated;

[0252] Question modality features are generated based on word vectors extracted from the questions to be answered.

[0253] In some embodiments, the fifth processing module may also be used for:

[0254] A 2D convolutional neural network is used to extract shape features and optical flow features from the video to be answered.

[0255] Based on shape features and optical flow features, a dense video summarization method is used to generate dense video summaries.

[0256] Based on dense video summarization, multiple summary representation vectors are generated by using word vector embedding and recurrent neural networks.

[0257] Multiple summary representation vectors are concatenated to generate summary modal features.

[0258] In some embodiments, the fifth processing module may also be used for:

[0259] The word embedding method is used to generate embedding vectors for each word in the question to be answered.

[0260] The embedding vectors corresponding to each word are concatenated to generate the problem modality features.

[0261] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an answer generation method for the video question-answering system. This method includes: independently modeling multiple acquired modal features to generate contextualized representations corresponding to each modal feature; determining the final layer node values ​​of the graph structure corresponding to each modal feature based on the contextualized representations; performing target round reasoning based on the final layer node values ​​of the graph structure corresponding to each modal feature to generate a final reasoning result; and generating a target answer based on the final reasoning result and the contextualized representations corresponding to each modal feature. The multiple modal features include video modal features corresponding to the video to be answered, question modal features corresponding to the question to be answered, and summary modal features corresponding to the video to be answered.

[0262] Further, the logic instructions in the memory 630 described above can be implemented by a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various media that can store program codes.

[0263] Further, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, when the computer program is executed by a processor, the computer can execute the answer generation method of the video question and answer system provided by the above-mentioned method embodiments, the method includes: respectively modeling the obtained multiple modal features, generating the situational representation corresponding to each modal feature; based on the situational representation corresponding to each modal feature, determine the final layer node value of the graph structure corresponding to each modal feature; based on the final layer node value of the graph structure corresponding to each modal feature, perform target round reasoning to generate a final reasoning result; based on the final reasoning result and the situational representation corresponding to each modal feature, generate a target answer; wherein, the multiple modal features include the video modal feature corresponding to the to-be-answered video, the question modal feature corresponding to the to-be-answered question and the summary modal feature corresponding to the to-be-answered video.

[0264] On the other hand, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the computer program realizes the answer generation method of the video question and answer system provided by the above-mentioned embodiments, the method includes: respectively modeling the obtained multiple modal features, generating the situational representation corresponding to each modal feature; based on the situational representation corresponding to each modal feature, determine the final layer node value of the graph structure corresponding to each modal feature; based on the final layer node value of the graph structure corresponding to each modal feature, perform target round reasoning to generate a final reasoning result; based on the final reasoning result and the situational representation corresponding to each modal feature, generate a target answer; wherein, the multiple modal features include the video modal feature corresponding to the to-be-answered video, the question modal feature corresponding to the to-be-answered question and the summary modal feature corresponding to the to-be-answered video.

[0265] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0266] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0267] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0268] The above embodiments are only used to illustrate the present application, and not to limit the present application. Although the present application has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications or equivalent replacements of the technical solutions of the present application do not deviate from the spirit and scope of the technical solutions of the present application, and should be covered in the scope of the claims of the present application.

Claims

1. A method for generating answers in a video question-and-answer system, characterized in that, include: The acquired modal features are modeled independently to generate contextualized representations for each modal feature. Based on the contextualized representations corresponding to each modal feature, the final layer node values ​​of the graph structure corresponding to each modal feature are determined; Target round reasoning is performed based on the final layer node values ​​of the graph structure corresponding to each modal feature to generate the final reasoning result; Based on the final reasoning result and the contextualized representations corresponding to each modality feature, a target answer is generated; The multiple modal features include the summary modal features corresponding to the video to be answered, the video modal features corresponding to the video to be answered, and the question modal features corresponding to the question to be answered generated based on dense video summarization; The determination of the final layer node values ​​of the graph structure corresponding to each modal feature based on the contextualized representation corresponding to each modal feature includes: Based on the contextualized representations corresponding to each modal feature, a graph structure corresponding to each modal feature is established respectively; Based on the graph structure, inference using a cross-modal attention mechanism is performed to generate the final layer node values ​​of the graph structure corresponding to each modal feature.

2. The answer generation method for the video question-answering system according to claim 1, characterized in that, The inference based on the graph structure using the cross-modal attention mechanism, generating the final layer node values ​​of the graph structure corresponding to each modal feature, includes: Based on the pairwise semantic correlation values ​​of all nodes in each of the graph structures, the initial node values ​​of the graph structures are updated using a graph convolutional neural network to generate the first node value; Using the target modal feature among the multiple modal features as the query modality, cross-modal attention is determined between the target modal feature and other modal features; Based on the cross-modal attention, the first node value of the graph structure corresponding to the target modal feature is updated, and the final layer node value of the graph structure corresponding to the target modal feature is generated.

3. The answer generation method for the video question-answering system according to claim 1, characterized in that, The number of nodes in the graph structure is equal to the length of the contextualized representation corresponding to the modal feature of the graph structure.

4. The answer generation method of the video question-answering system according to any one of claims 1-3, characterized in that, The target round inference is performed based on the final layer node values ​​of the graph structure corresponding to each modal feature, generating the final inference result, including: Based on the final layer node values ​​of the graph structure corresponding to each modality, determine the problem-guided modal features corresponding to each modality in the current round of reasoning; Merge the modal features guided by the questions corresponding to each modality under the current round of reasoning to determine the first reasoning result under the current round of reasoning; If the current round number is less than the target round, then based on the first reasoning result under the current round reasoning, generate the first reasoning result under the next round reasoning. The current round number is determined to be equal to the target round, and the first reasoning result under the current round reasoning is determined as the final reasoning result.

5. The answer generation method of the video question-answering system according to any one of claims 1-3, characterized in that, The generation of the target answer based on the final reasoning result and the contextualized representations corresponding to each modality feature includes: The final inference result, the contextualized representation corresponding to the summary modality feature, and the contextualized representation corresponding to the video modality feature are merged to generate the final representation; The target answer is generated based on the question type of the question to be answered and the final representation.

6. The answer generation method for the video question-answering system according to claim 5, characterized in that, The process of generating the target answer based on the question type of the question to be answered and the final representation includes: The question type is determined to be a selection question. The final representation is then subjected to classification task processing to generate the target answer. The question type is determined to be a numerical question. The final representation is then subjected to regression task processing to generate the target answer.

7. The answer generation method of the video question-answering system according to any one of claims 1-3, characterized in that, Before independently modeling the acquired multiple modal features and generating contextualized representations corresponding to each modal feature, the method includes: The summary modal features are generated based on the shape features and optical flow features extracted from the video to be answered; Based on the shape features and motion features extracted from the video to be answered, the video modal features are generated; The question modality features are generated based on the word vectors extracted from the question to be answered.

8. The answer generation method for the video question-and-answer system according to claim 7, characterized in that, The step of generating the summary modal features based on the shape features and optical flow features extracted from the video to be answered includes: The shape features and optical flow features are extracted from the video to be answered using a 2D convolutional neural network. Based on the aforementioned shape features and optical flow features, a dense video summarization method is used to generate a dense video summary. Based on the aforementioned dense video summarization, multiple summary representation vectors are generated using word vector embedding and recurrent neural networks. The multiple summary representation vectors are concatenated to generate the summary modal features.

9. The answer generation method for the video question-answering system according to claim 1, characterized in that, The step of generating the question modality features based on the word vectors extracted from the question to be answered includes: The word vector embedding method is used to generate embedding vectors for each word in the question to be answered. The embedding vectors corresponding to each word are concatenated to generate the problem modality features.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the answer generation method of the video question-answering system as described in any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the answer generation method of the video question-answering system as described in any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the answer generation method of the video question-answering system as described in any one of claims 1 to 9.

Images