A fine-grained video emotion content question answering method and system based on multimodal data

Abstract

This invention belongs to the field of video question answering, specifically relating to a fine-grained video emotion content question answering method and system based on multimodal data. This invention constructs a baseline model for video emotion inference based on an episodic memory network, designs multi-branch processing modules for visual, audio, and text data, and utilizes a Transformer encoder to encode temporal dependencies in multimodal data. The extracted multimodal features contain multi-faceted emotional content, enabling accurate completion of fine-grained video emotion content question answering tasks. This invention utilizes a Transformer encoder to learn temporal relationships on video, audio, and text sequences, and extracts high-dimensional multimodal features related to emotion classification. These temporal relationships are crucial for analyzing the emotional information contained in videos. This invention can effectively improve the accuracy of fine-grained video emotion content question answering tasks based on multimodal data.

Description

Technical Field

[0001] This invention belongs to the field of video question answering, specifically relating to a fine-grained video emotion content question answering method and system based on multimodal data. Background Technology

[0002] In recent years, sentiment analysis in movies and television programs has received increasing attention in fields such as affective computing and artificial intelligence system design. Movies and television programs contain rich interactive scenes and character relationships, allowing characters to experience emotions similar to those felt by humans in the real world, such as excitement from rewards and sadness from separation. The human-centered video scenes are closely related to real-life social scenarios, providing a platform for training AI systems to understand the high-level semantic information behind emotions—the reasons for emotions, intentions, and human motivations. The rich emotions contained in videos provide data support for studying the emotional content inherent in the video content itself.

[0003] Intelligent systems need to possess the ability to understand video scenes with fine granularity, not only identifying emotion categories but also reasoning about the underlying reasons, intentions, and user motivations in an interpretable manner. In research on the emotional content contained in videos, multimodal information from videos is often used as a source of analytical data, such as in video emotion recognition of characters. Multimodal video emotion recognition methods primarily understand the emotions conveyed by video content through audio, text, and visual information. For example, based on video context information and facial expression information of characters in the video, a cascaded structured model composed of RNNs is used to identify emotions in videos; by integrating facial expression features of characters in the video and video background information, a two-stream coding model is used to identify video emotions; based on the interaction between visual content and text vocabulary, a collaborative memory network is used to identify emotions in multimodal information; and based on multimodal transformation networks, visual and auditory representations are mapped to the same feature space for video emotion recognition. The above-mentioned research on video emotion understanding technologies mainly focuses on video emotion recognition methods, with relatively little research on emotion reasoning within videos. With the development of intelligent interactive applications, researchers have begun to explore methods for inferring the underlying causes of emotions based on multimodal scenarios. For example, they are extracting emotion-cause pairs from multimodal information in video dialogues and inferring the reasons behind emotions through understanding this multimodal information. Therefore, building upon video emotion recognition based on multimodal information, further research is needed on video emotion inference methods to support a deeper understanding of the reasons that induce emotions in multimodal video content.

[0004] To enhance the intelligence of human-computer interaction, in addition to recognizing video emotions, it is also necessary to correctly understand the user's emotional needs and intentions. This is precisely the main research content of video emotion reasoning. Therefore, in-depth research on video emotion understanding technology needs to further explore how to use interpretable methods to reason about emotions in video interaction scenarios, that is, to understand the reasons, intentions, and user behavioral motivations behind the video. Video emotion reasoning needs to be conducted in human-centered video interaction scenarios. Only by placing people in specific contexts can we fully utilize the multimodal information in the interaction process to understand emotions and then reason about the multiple aspects involved in emotions.

[0005] 1) Existing technologies: Emotional cause extraction and emotion inference

[0006] Discovering the underlying causes of specific emotional expressions within the context of dialogue text or in multimodal scenarios has always been a hot topic in the field of affective computing. Emotion cause extraction is a refined task of affective analysis, aiming to explore the underlying causes of certain emotional expressions in dialogue. Rui Xia et al. (Reference: Xia R, Zhang M, Ding Z. RTHN: A rnn-transformer hierarchical network for emotion cause extraction[C] / / Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019.ijcai.org, 2019:5285-5291.) considered the relationships between multiple clauses in a dialogue and used them for affective cause extraction. Since dialogues in videos are inherently composed of multimodal information, Rui Xia et al. (Reference: Wang F, Ding Z, Xia R, et al. Multi modal emotion-cause pair extraction in conversations[J]. CoRR, 2021, abs / 2110.08020.) further proposed a multimodal emotion-cause pair extraction task, jointly extracting emotions and their triggering causes from the multimodal information of the dialogue. MECPE is a further exploration of the ECE task, requiring the model to have strong video multimodal information understanding and emotion inference capabilities, because the causes of emotions do not necessarily come only from textual information, but may also come from the visual scene.

[0007] Furthermore, opinion mining is a crucial issue in text sentiment analysis, with sentiment inference being a subtask within text sentiment analysis addressing the question of "who holds the opinion and why." While research on video sentiment inference tasks within the field of video sentiment understanding is currently limited, the aforementioned work demonstrates the profound research significance and application value of inferring sentiment from videos containing multimodal information. Therefore, there is an urgent need for an interpretable method to infer emotions within videos, fully understanding the reasons, intentions, and behavioral motivations that evoke emotions in multimodal video content.

[0008] 2) Existing technology: Multimodal sentiment analysis

[0009] Multimodal sentiment analysis in videos aims to understand the emotions conveyed by video content through audio, text, and visual information. For example, Sun Man-Chin et al. (Reference: Sun MC, Hsu SH, Yang MC, et al. Context-aware cascade attention-based RNN for video emotion recognition[C] / / 2018 First Asian Conference on Affective Computing and Intelligent Interaction(ACIIAsia). IEEE, 2018: 1-6.) proposed a cascaded structure model consisting of two RNNs, which uses video context information and facial expression information of people in the video to identify emotions in the video. Lee Jiyoung et al. (Reference: Lee J, Kim S, Kim S, et al. Context-aware emotion recognition networks[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 10143-10152.) designed a deep network to integrate facial expression features of people in the video and video background information in a joint and boosting manner to identify video emotions. Nan Xu et al. (Reference: Xu N, Mao W, Chen GA co-memory network for multimodal sentiment analysis[C] / / The 41st International ACM SIGIR Conference on Research & Development in Information Re-trieval,SIGIR2018,Ann Arbor,MI,USA,July 08-12,2018.ACM,2018:929-932.) used a co-memory network to iteratively model the interaction between visual content and text vocabulary for multimodal sentiment analysis.Qi Fan et al. (Reference: Qi F, Yang X, Xu C. Zero-shot video emotion recognition via multimodalprotagonist-aware transformer network[C] / / MM'21:ACM Multimedia Conference,Virtual Event,China,Oc-tober 20-24,2021.ACM,2021:1074-1083.) proposed a multimodal transformation network that maps visual and auditory representations to the same feature space for video emotion recognition.

[0010] However, the aforementioned multimodal sentiment analysis research mainly focuses on emotion recognition tasks supervised by emotion labels. In recent years, some works have begun to focus on a deeper understanding of video emotions. Unlike the direct identification of the emotion categories of characters in videos, Guangyao Shen et al. (Reference: Shen G, Wang X, Duan X, et al. Memor: A dataset for multimodal emotion reasoning in videos[C] / / MM'20:The 28th ACM International Conference on Multimedia,Virtual Event / Seattle,WA,USA,October12-16,2020.ACM,2020:493-502.) proposed character emotion reasoning based on video multimodal information. Addressing the problem of emotion recognition for characters lacking multimodal information in video interaction scenarios, they proposed using the emotions of other characters in the same scene to infer their emotions. This process does not involve understanding the underlying reasons, intentions, or behavioral motivations of the emotions. Zadeh et al. (Reference: Zadeh A, Chan M, Liang PP, et al. Social-iq: A question-answering benchmark for artificial social intelligence[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2019:8807-8817.) constructed the Social-IQ dataset and labeled some question-answer pairs with sentiment or emotional information to support social intelligence systems in understanding users' psychological states by reasoning about the answers to questions. As can be seen, based on multimodal sentiment analysis, inferring video emotions based on multimodal information aligns with the development trend of multimodal sentiment analysis and is a key area for further exploration.

[0011] 3) Existing technology: Question answering technology

[0012] Unlike traditional emotion recognition based on sentiment labels, artificial intelligence systems need to go beyond sentiment label-supervised emotion recognition and achieve the ability to infer emotions in videos in an interpretable way. Question answering technology is an interpretable method for exploring the degree of understanding of underlying phenomena. Question answering technology has been used in various research fields and has achieved good results, such as natural language processing, vision and language, and commonsense reasoning. Video question answering is a task of answering questions about a given video in natural language format. Due to its applicability in fields such as social intelligence systems and cognitive robots, it has attracted widespread attention in the past few years. Currently, there are various methods in the field of video question answering, including attention mechanisms, multimodal fusion methods, dynamic memory networks, and multimodal relation learning. Good results have been achieved based on public video question answering datasets. Therefore, this invention chooses to rely on question answering tasks to realize the model's inference of video emotions. Summary of the Invention

[0013] Video emotion reasoning is currently a hot research topic in the field of video emotion understanding. Interpretable reasoning based on question-and-answer formats provides an efficient method for understanding emotions in videos. Generally, the reasons, intentions, and behavioral motivations behind emotions in videos are contained within the video's multimodal data. Therefore, it is necessary to achieve a fine-grained understanding of video scenes and extract useful information for emotion reasoning from multimodal information. In typical emotion reasoning work based on video multimodal information, a single reasoning step is often insufficient to learn enough effective information; therefore, multi-step reasoning is key to improving model performance.

[0014] To solve the above problems, the technical solution proposed by this invention is as follows:

[0015] A fine-grained video sentiment content question answering method based on multimodal data, comprising the following steps:

[0016] 1) Divide the long video into segments based on a number of dialogue sentences, and then divide the corresponding subtitle text and audio to obtain several video clips;

[0017] 2) Extract multimodal features from a video segment, including visual features, audio features, and text features, and encode the corresponding question-answer pairs to obtain the question code q. T and answer codes

[0018] 3) Perform temporal coding on the extracted multimodal features respectively;

[0019] 4) Extract question-related information from the multimodal features of the video based on the visual branch, audio branch and text branch. Enhance the visual branch with the facial features of the people in the video and enhance the text branch with the story plot information in the video story summary information to obtain the enhanced multimodal features.

[0020] 5) The enhanced multimodal features are input into the episodic memory network. The episodic memory network is used to update and store the emotional reasoning cues extracted from the multimodal features, capturing key multimodal information in the emotional reasoning process to obtain the video context representation C. v,a,t ;

[0021] 6) Encode the above problem q T and answer codes and video context representation C v,a,t The input answer prediction module learns context-aware attention for both question and answer encodings to obtain the final sentiment-based question answering result P.

[0022] Furthermore, the segmentation of the long video into units of several lines of dialogue is actually segmenting the long video into units of n lines of dialogue, where n≤20.

[0023] Furthermore, the visual features include global visual features and facial features.

[0024] Furthermore, methods for extracting global visual features include using a ResNet-152 model pre-trained on the ImageNet dataset.

[0025] Furthermore, the method for extracting facial features includes: using the pre-trained model MTCNN to detect facial regions in video frames; fine-tuning the VGGFace2 model pre-trained on Facenet using facial region data of the main characters in the video (e.g., the 6 main characters in the Friends video) to obtain facial recognition features; extracting facial expression features using the Facenet model pre-trained on the FER2013 dataset; and concatenating the facial recognition features and facial expression features to form the facial features of the person in the video.

[0026] Furthermore, methods for extracting audio features include using the openSIMLE audio feature extractor.

[0027] Furthermore, methods for extracting text features and encoding question-answer pairs include using pre-trained GloVe word embedding tools.

[0028] Furthermore, methods for encoding multimodal information include using a transformer encoder.

[0029] Furthermore, the extraction of question-related information from the multimodal features of the video is achieved by using question-guided attention to obtain question-related feature representations through the following steps:

[0030] 1) Facial features With question code q TPerform a dot product to obtain the similarity s between the question and the feature;

[0031] 2) Use the softmax function to process the dot product result s to obtain a representation of facial features. Spatial attention a f ;

[0032] 3) Spatial attention a f With facial features Perform a dot product to obtain the feature representations relevant to the problem.

[0033] Furthermore, the video context representation C output by the episodic memory network is obtained through the following steps. v,a,t :

[0034] 1) Attention mechanism: Calculate the gate mechanism attention score for t update processes. Where F attn Represents the attention function (Reference: Xiong C, Merity S, Socher R.Dynamic memory networks for visual and textual question answering[C] / / JMLR Workshop and ConferenceProceedings:volume 48Proceedings of the 33nd International Conference onMachine Learning,ICML 2016,New York City, NY, USA, June 19-24,2016.JMLR.org,2016:2397-2406.), f i This indicates the output of the i-th fact vector in the real-time sequence, m t-1 It represents the state after the (t-1)th update in the memory network module, and q represents the problem encoding vector;

[0035] 2) Memory unit update mechanism: Calculate the hidden state of the i-th unit of the GRU (gated recurrent unit) in the memory network module. Where h i Let represent the hidden state of the i-th unit in the GRU, and let the last hidden state of the GRU serve as the video multimodal context representation for the t-th memory unit update. The subscript t represents the t-th update, and the superscript t represents the text modality; finally, the memory unit state is updated for the t-th update. Where F mem It is a memory update function.

[0036] Furthermore, the video question-and-answer results of the answer prediction module are obtained through the following steps:

[0037] 1) Use the context matching module (Reference: Seo MJ, Kembhavi A, Farhadi A, et al. Bidirectional attention flow for machine comprehension[C] / / 5th International Conference on Learning Representations,ICLR 2017,Toulon,France,April 24-26,2017,Conference Track Proceedings.OpenReview.net,2017.) to compute the fused representation of each modality feature with the question and answer representations. Where E m The modal features output by the memory network module, q v,a,t It is a context-aware question representation. It represents the answer based on contextual awareness.

[0038] 2) Use the softmax function and FC layer to process the fused representation and obtain the probability distribution of the predicted answer for each branch.

[0039] 3) Concatenate the answer prediction distributions obtained from the three modalities, and process them using the linear and softmax functions to obtain the final answer prediction probability distribution P = softmax(linear([P v ;P a ;P t ])).

[0040] Furthermore, steps 3)-6) are implemented using a video emotion content question answering model, wherein the video emotion content question answering model is trained end-to-end, and the loss function of the video emotion content question answering model is... Here, the predicted distribution of the five candidate answers is given, and each element p in P = [p0, ..., p4] is... i This indicates that the answer corresponding to the sample is a. i The probability; y = [y0, ..., y4] is the one-hot encoded representation of the sample label, when the answer corresponding to the sample is a. i time y i =1, otherwise y i =0.

[0041] A fine-grained video sentiment content question answering system based on multimodal data, comprising:

[0042] The video segmentation module is used to segment long videos into units of several lines of dialogue, and to segment the corresponding subtitle text and audio to obtain several video segments;

[0043] The multimodal feature extraction module is used to extract multimodal features from a video clip, including visual features, audio features, and text features, and to encode the corresponding question-answer pairs to obtain question codes and answer codes;

[0044] The encoding module is used to perform temporal encoding on the extracted multimodal features respectively;

[0045] The multimodal feature enhancement module is used to extract question-related information from the multimodal features of a video based on visual, audio, and text branches. It enhances the visual branch using facial features of people in the video and enhances the text branch using story plot information from the video's synopsis, thus obtaining enhanced multimodal features.

[0046] The Episode Memory Network module is used to input the enhanced multimodal features into the Episode Memory Network, and use the Episode Memory Network to update and store the emotional reasoning cues extracted from the multimodal features, capture key multimodal information in the emotional reasoning process, and obtain the video context representation.

[0047] The answer prediction module is used to learn context-aware attention for question codes and answer codes respectively, taking question codes, answer codes and video context representations as input, to obtain the final sentiment question answering results.

[0048] A computer device includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0049] A computer-readable storage medium storing a computer program that, when executed by a computer, implements the methods described above.

[0050] In summary, compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0051] 1. This invention constructs a baseline model for video emotion reasoning based on Episode Memory Network, designs multi-branch processing modules for visual, audio, and text data, and uses a Transformer encoder to encode temporal dependencies in multimodal data. The extracted multimodal features contain multi-angle emotional content, which can accurately complete fine-grained video emotion content question answering tasks.

[0052] 2. This invention utilizes a transformer encoder to learn temporal relationships in video, audio, and text sequences, and extracts high-dimensional multimodal features related to emotion classification. These temporal relationships are crucial for analyzing the emotional information contained in videos.

[0053] 3. Based on the human attention to facial and story information during emotional reasoning, this invention uses a question-guided attention mechanism in the visual and text branches to focus on facial expression features in visual features and story information in text features, and extracts story summary information from text features, effectively improving the reasoning accuracy and question-answering accuracy of the model method.

[0054] 4. This invention addresses the problem of limited effective information learned through single-step reasoning by accumulating multimodal cues for reasoning through a multi-step update process of memory units via an episodic memory network. These cues are then used to predict the final answer to emotion-related questions, effectively improving the results of fine-grained video emotion content question answering tasks based on multimodality. Attached Figure Description

[0055] Figure 1 A diagram illustrating a multimodal, fine-grained video emotion-based question-and-answer task.

[0056] Figure 2 A flowchart illustrating the network framework of a fine-grained video emotion content question answering method based on multimodal data. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the following detailed description of the fine-grained video emotion content question-answering method for multimodal data provided by the present invention is provided in conjunction with the accompanying drawings, but does not constitute a limitation of the present invention.

[0058] Please refer to Figure 1 , Figure 2 This invention proposes a fine-grained video emotion content question answering method based on multimodal information. This method extracts question-related features from video multimodal information based on visual, audio, and text branches; uses question-guided attention to extract facial expression features from visual features and story summary information from text features; achieves multi-step emotion reasoning by iteratively updating the plot memory storage unit to capture emotion-related cues, and utilizes a Transformer encoder to learn the temporal dependencies of multimodal data; uses a question-guided attention mechanism to extract facial expression information from the visual branch and story plot information from the text branch; and finally accumulates emotion reasoning cues through multi-step reasoning via the plot memory network, learning effective video context representations for answer prediction.

[0059] 1. Multimodal feature extraction

[0060] 1) Visual feature extraction

[0061] First, video frames are extracted from the video at 3fps for subsequent visual feature extraction. This invention mainly focuses on extracting two types of visual features from video frame data: global visual features of the video frame and facial features.

[0062] Global visual features: Video frames are processed using a ResNet-152 model pre-trained on the ImageNet dataset to obtain visual features of video frames with a feature dimension of 204. Then, the visual features of video frames corresponding to a video segment are stacked to obtain the visual features representing the entire video segment. Where n clp This indicates the number of video frames in a video segment.

[0063] Facial Features: First, the pre-trained MTCNN model is used to detect facial regions in video frames. Based on the facial regions of the characters in the video, two main types of features are extracted: facial recognition features and facial expression features. For example, facial recognition features are obtained by fine-tuning the VGGFace2 model pre-trained on FaceNet using facial region data of the six main characters in the "Friends" video; facial expression features are extracted using a FaceNet model pre-trained on the FER2013 dataset. Finally, the facial recognition features and facial expression features are concatenated to form the facial features of the characters in the video. Where n f This indicates the number of facial regions in each video clip.

[0064] The input data for the visual branch in this method is V. clp and V f .

[0065] 2) Audio Feature Extraction

[0066] First, to align the audio data with the subtitle text, the audio data corresponding to each video segment is segmented into 20 segments based on the timestamps of the subtitle text, supporting the model's learning of the contextual relationships between audio segments. Using the openSIMLE feature extractor, guided by the ComParE_2016 configuration file, 6373-dimensional acoustic features are extracted from the 20 audio segments corresponding to each video segment. Finally, the acoustic features corresponding to each video segment can be represented as... Where n a This indicates the number of audio segments corresponding to a video clip.

[0067] The input data for the audio branch in this method is A.

[0068] 3) Text Feature Extraction

[0069] First, character vector features are extracted from the input text data. Then, GloVe, trained on Wikipedia2014 and Gigaword5, is used to obtain 300-dimensional word encoding features, resulting in the feature representation S∈ for the subtitle text corresponding to each video segment. Where n set n represents the number of subtitle text sentences corresponding to a video clip. wrd This represents the number of words in each sentence. Additionally, the story synopsis text feature corresponding to each video segment can be represented as... Where n ks n represents the number of sentences in the synopsis corresponding to each video segment. kw This indicates the number of words in each sentence.

[0070] The input data for the text branch in this method are S and K.

[0071] 2. Transformer-based multimodal information encoding

[0072] Transformer excels in numerous natural language processing problems, demonstrating its ability to learn long-range data dependencies. The Transformer's ability to learn long-term dependencies in sequential data primarily relies on its self-attention mechanism. Furthermore, to learn information from sequential data in different feature subspaces, this invention employs a multi-head attention mechanism within the Transformer, implemented through parallel processing of multiple self-attention mechanisms, as expressed in the formula:

[0073] MHA(Q,K,V)=Concat(head1,…,head k W 0

[0074]

[0075]

[0076] Where Q represents the query matrix, K represents the key matrix, V represents the value matrix, W is the weight matrix, and Attention(·) represents the self-attention mechanism calculation process. i This represents the i-th "attention head" in the multi-head attention mechanism. This method uses three independent Transformer encoders to perform temporal encoding on the sequential input data of each branch. The number of Transformer encoder layers is 2, and the number of "attention heads" in the multi-head attention mechanism is 6.

[0077] For the vision branch, the input data includes Vclp and V f Before further encoding the input data, a linear transformation layer is first used to process the visual features V. clp and V f By unifying the two visual feature dimensions to 300, the linearly transformed visual features are obtained. and Then, and The data are used as input to the Transformer encoder to learn the temporal dependencies of the video visual sequence features. Then, the output of the last layer of the Transformer encoder is used as the visual features obtained through further encoding. and

[0078] For the audio branch, the same method is used to obtain further encoded features of the audio data through linear transformation and Transformer encoding.

[0079] For text branches, the same method is used to obtain text features through linear transformation and Transformer encoding. and For the question text features and answer text features in the data, separate Transformer encoders are used for encoding to obtain the question text representation. and answer text representation Where, n q It is the number of words in the question text statement. The answer text statement a iT The number of words in the text.

[0080] 3. Problems guide attention

[0081] Inspired by the cognitive process of reasoning about the underlying causes of emotions, this invention uses facial features of people in videos to enhance visual branches and storyline information from video synopsis to enhance text branches. Facial expressions are a direct reflection of human emotions; when understanding and inferring emotions in videos, people tend to focus on the facial expressions of people in the video. The first step in using facial expressions to enhance visual branches is to extract question-related information from facial features using attention mechanisms. Taking visual branches as an example, the question-guided attention mechanism is as follows:

[0082]

[0083] a f =softmax(s)

[0084]

[0085] Where s represents the similarity between the question and the facial features, a f Indicate facial features The spatial attention score is used to extract problem-related features to obtain facial feature representation. The visual features of the enhanced visual branch can be represented as in n v is the number of video frames corresponding to the video segment; represents the connection symbol.

[0086] 4. Multimodal Episodic Memory Network

[0087] Episodic memory networks are designed to retrieve question-related information from a sequence of facts input to answer questions, and are particularly suitable for questions that require reasoning based on video context. This invention utilizes episodic memory networks to update and store emotional reasoning cues extracted from multimodal features.

[0088] In this invention, the memory network modules corresponding to the three independent branches have the same structure as the aforementioned plot memory network. Each memory network module requires three updates, and each modal feature is mapped to an input fact representation matrix F.

[0089] Visual-M is the visual memory network module corresponding to the visual branch. First, visual features are organized into a visual fact matrix that is input into the visual memory network module. Video segments are organized into processing units of 10 seconds each. Since video frame extraction uses 3fps, dividing the video segment into 30-frame segments yields the video visual representation V. s = Among them, s i Let n represent the i-th segment. s This indicates the number of segments the video is divided into. Therefore, Let f represent the visual fact matrix input to the visual memory network module, and f i =s i After processing by the visual memory network module, the final visual context representation can be obtained. It contains visual information related to the problem.

[0090] Audio-M is the audio memory network module corresponding to the audio branch. It extracts problem-related features from the audio features through multi-step updates. The audio features are obtained by encoding the audio features in the audio branch using a Transformer encoder. As the input to the audio memory network module, the audio fact matrix of the audio memory network module is: Where n a This refers to the number of audio segments. Through a process similar to that used in the visual memory network module, the last updated state of the audio memory network module can be obtained. As the final audio context representation

[0091] Textual-M is a text memory network module corresponding to the text branch, designed to learn and store sentiment information in text features by learning the interaction between text features and question encodings. Text representation in the text branch. The text fact matrix F serves as the input to the text memory network. t , where sn i Table text sentence vector representation, l t This represents the number of sentences in the text features. After multiple updates of the text features related to the question by the text memory network, the final output is a text context representation. Used for answer prediction.

[0092] 5. Answer Prediction

[0093] The answer prediction module aims to jointly model the video's multimodal feature representation and question-answer pair encoding to predict the answer to the question. The core of the answer prediction module is the context matching module. The context matching module will convert the final multimodal context representation C... v,a,t Problem code q T and answer codes As input, context-aware attention is learned for both question and answer encodings, respectively, to obtain a context-aware question representation q. v,a,t and context-aware answer representation Since the working principle of the context matching module is the same in the three branches of this invention, only the video modal features it processes are different, the working process is described using the context matching module in the vision branch as an example.

[0094] The output of the visual memory network module serves as the visual feature input to the context matching module of the visual branch. After calculating the visual feature perception attention encoded in the question, a visual perception question representation can be obtained. Furthermore, by calculating the visual feature perception attention for the answer encoding, a visual perception answer representation can be obtained.

[0095] Finally, the visual branch answer prediction module predicts visual features C. v The problem is represented by q. v and answer representation The fusion is represented as follows:

[0096]

[0097] in, This represents element-wise multiplication. The probability distribution for the visual branch's answer prediction is calculated as follows:

[0098]

[0099] in, For both the audio and text branches, the same calculation process can be used to obtain the predicted probability distribution of the answer for the audio branch. The probability distribution of the predicted answer based on the text branch The probability distribution of the final answer prediction is calculated as follows:

[0100] P = softmax(linear([P v ;P a ;P t ]))

[0101] Used for predicting the final answer.

[0102] 6. Training and validation of a fine-grained video sentiment content question answering model based on multimodal data

[0103] Furthermore, the fine-grained video sentiment content question answering deep learning model based on multimodal data was trained and validated. The loss function of this model is:

[0104]

[0105] The overall training objective of this model is:

[0106]

[0107] Among them, X R θ represents all the sample data in the entire dataset. v ,θ a ,θ t These are the parameters for the visual branch, audio branch, and text branch, respectively. The parameter θ is updated by modifying the visual branch, audio branch, and text branch. v ,θ a ,θ t .

[0108] Another embodiment of the present invention provides a fine-grained video emotion content question-answering system based on multimodal data, comprising:

[0109] The video segmentation module is used to segment long videos into units of several lines of dialogue, and to segment the corresponding subtitle text and audio to obtain several video segments;

[0110] The multimodal feature extraction module is used to extract multimodal features from a video clip, including visual features, audio features, and text features, and to encode the corresponding question-answer pairs to obtain question codes and answer codes;

[0111] The encoding module is used to perform temporal encoding on the extracted multimodal features respectively;

[0112] The multimodal feature enhancement module is used to extract question-related information from the multimodal features of a video based on visual, audio, and text branches. It enhances the visual branch using facial features of people in the video and enhances the text branch using story plot information from the video's synopsis, thus obtaining enhanced multimodal features.

[0113] The Episode Memory Network module is used to input the enhanced multimodal features into the Episode Memory Network, and use the Episode Memory Network to update and store the emotional reasoning cues extracted from the multimodal features, capture key multimodal information in the emotional reasoning process, and obtain the video context representation.

[0114] The answer prediction module is used to learn context-aware attention for question codes and answer codes respectively, taking question codes, answer codes and video context representations as input, to obtain the final sentiment question answering results.

[0115] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0116] Experimental data: The results of comparing the method proposed in this invention with other methods are shown in Table 1.

[0117] Table 1

[0118] method Image modality audio modality Text modality Accuracy (%) 1 Random - - - 20.00 2 Longest Answer - - - 32.24 3 Shortest Answer - - - 16.27 4 HRCN P - - 47.41 5 HGA P - P 57.99 6 Two-stream P P - 59.90 7 Two-stream P - P 58.46 8 Two-stream - P P 58.59 9 Two-stream P P P 61.16 10 Invention Method - Dual-modal P P - 61.88 11 Invention Method - Dual-modal P - P 69.07 12 Invention Method - Dual-modal - P P 58.91 13 Method of the present invention P P P 65.62

[0119] Another embodiment of the present invention provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0120] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the methods described above.

[0121] The fine-grained video emotion content question answering algorithm based on multimodal data described above has been explained in detail. However, it is obvious that the specific implementation of the present invention is not limited thereto. For those skilled in the art, various obvious modifications made to the method without departing from the spirit and scope of the claims are within the protection scope of the present invention.

Claims

1. A fine-grained video emotion content question-answering method based on multi-modal data, characterized in that, The method comprises the following steps: 1) segmenting a long video in units of several dialogues, and segmenting corresponding subtitle text and audio to obtain several video segments; 2) extracting multi-modal features including visual features, audio features and text features for a video segment, and encoding the corresponding question-answer pair to obtain question encoding and answer encoding; 3) separately performing time sequence encoding on the extracted multi-modal features; 4) extracting question-related information from the multi-modal features of the video based on the visual branch, the audio branch and the text branch, enhancing the visual branch using the facial features of the characters in the video, and enhancing the text branch using the plot information in the video story outline to obtain enhanced multi-modal features; 5) inputting the enhanced multi-modal features into a plot memory network, updating and storing the emotional reasoning clues extracted from the multi-modal features using the plot memory network, capturing multi-modal key information in the emotional reasoning process, and obtaining a video context representation; 6) inputting the question encoding and the answer encoding and the video context representation into an answer prediction module to learn context-aware attention for the question encoding and the answer encoding respectively, and obtaining a final emotional question-answer result; wherein step 5) obtains the video context representation C output by the episodic memory network by the following steps v,a,t : 1) Attention mechanism: compute the gating mechanism attention score for the t-th update process where F attn denotes the attention function, f i denotes the i-th fact vector in the output real-time sequence, m t-1 is the state after the t-1-th update in the memory network module, and q denotes the question encoding vector; 2) Memory cell update mechanism: Compute the hidden state of the i-th cell of the GRU in the memory network module where h i denotes the hidden state of the i-th cell in the GRU, and the last layer hidden state of the GRU as the video context representation of the t-th memory cell update Finally, update the t-th memory cell state where F mem is the memory update function.

2. The method of claim 1, wherein, The visual features include global visual features and facial features; The global visual features are extracted using a Resnet-152 model pre-trained on an ImageNet dataset; The method for extracting facial features includes: detecting facial regions in video frames using a pre-trained model MTCNN, fine-tuning a VGGFace2 model pre-trained on Facenet using facial region data of main characters in the video to obtain facial recognition features, and extracting facial expression features using a Facenet model pre-trained on a FER2013 dataset; and splicing the facial recognition features and the facial expression features to obtain facial features of the characters in the video.

3. The method of claim 1, wherein, The audio features are extracted using an openSIMLE audio feature extractor, the text features are extracted and the question-answer pair is encoded using a pre-trained GloVe word embedding tool method, and the extracted multi-modal features are separately time sequence encoded using a transformer encoder.

4. The method of claim 1, wherein, The question-related information is extracted from the multi-modal features of the video using question-guided attention to obtain question-related feature representations, which comprises the following steps: 1) multiply the face features with the problem code q T to get the similarity s between the problem and the features; 2) Process the dot product result s with a softmax function to get the spatial attention a representing the facial features f ; 3) spatial attention a f with features dot product with features 5. The method of claim 1, wherein, The video question-answering result of the answer prediction module is obtained by the following steps: A context matching module is used to calculate the fusion representation of each modal feature and the question and answer representation; A softmax function and an FC layer are used to process the fusion representation to obtain the probability distribution of answer prediction of each branch; The probability distributions of answer prediction obtained by the three modalities are spliced, and a linear and softmax function is used for processing to obtain the final answer prediction probability distribution.

6. The method of claim 1, wherein, Steps 3) to 6) are implemented using a video emotional content question answering model, wherein the video emotional content question answering model is trained end-to-end, and a loss function of the video emotional content question answering model P = [p0,..., p4] is a probability distribution over the answer set A = {a0,..., a4} i , where each element p in P represents a probability that the answer corresponding to the sample is a i ; y = [y0,..., y4] is a one-hot encoding representation of the sample label, where y i = 1 when the answer corresponding to the sample is a i , otherwise y i = 0.

7. A multimodal data based fine-grained video emotion content question answering system using the method of any one of claims 1-6. The method comprises the following steps: A video segmentation module is configured to segment a long video in units of several dialogues, and segment corresponding subtitle text and audio to obtain several video segments; The multi-modal feature extraction module is configured to extract multi-modal features including visual features, audio features and text features for a video clip, and encode corresponding question-answer pairs to obtain question encoding and answer encoding. The encoding module is configured to perform time sequence encoding on the extracted multi-modal features respectively. The multi-modal feature enhancement module is configured to extract question-related information from the multi-modal features of the video based on a visual branch, an audio branch and a text branch, enhance the visual branch using facial features of a character in the video, enhance the text branch using plot information in a video story outline, and obtain enhanced multi-modal features. The plot memory network module is configured to input the enhanced multi-modal features into a plot memory network, update and store emotional reasoning clues extracted from the multi-modal features using the plot memory network, capture multi-modal key information in an emotional reasoning process, and obtain a video context representation. The answer prediction module is configured to input the question encoding and the answer encoding and the video context representation as inputs, learn context-aware attention for the question encoding and the answer encoding respectively, and obtain a final emotional question-answer result.

8. A computer device, comprising: The computer readable storage medium stores a computer program configured to be executed by the processor, and the computer program includes instructions for executing the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program configured to be executed by the processor, and the computer program includes instructions for executing the method in any one of claims 1-6.

Images