A rich semantic video dialogue generation method based on hierarchical visual attention
By segmenting videos into segments and utilizing a hierarchical visual attention mechanism to extract image and motion features, the problem of existing models ignoring fine-grained information is solved, achieving high-quality video dialogue generation that is applicable to multimodal information channels.
Patent Information
- Application Number
- CN202311558183.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-21
- Publication Date
- 2026-01-09
- Estimated Expiration
- 2043-11-21
AI Technical Summary
Existing video dialogue generation models ignore the unique temporal and spatial features of videos, resulting in an inability to effectively capture important fine-grained information, and poor dialogue quality and performance stability in different scenarios.
The input video is segmented into video clips, and C3D-rgb and C3D-flow models are used to extract visual and motion features. Audio features are extracted by combining the VGG model. The question is analyzed word by word through the attention memory unit to generate dialogue responses. A hierarchical visual attention mechanism is used to fuse visual and motion features.
It improves the accuracy and efficiency of video dialogue generation, effectively captures fine-grained information, generates semantically rich and coherent dialogue content, is applicable to multiple modal information channels, and improves the scalability of the model and the quality of dialogue.
Smart Images
Figure CN117668288B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The embodiment of the present disclosure relates to the technical field of multi-modal human-computer interaction and video dialogue generation, in particular to a rich semantic video dialogue generation method based on hierarchical visual attention. BACKGROUND
[0002] Video dialogue is an important content in the field of multi-modal human-computer interaction. Video dialogue contains a large amount of spatio-temporal visual information and complex multi-modal relationships, which brings great challenges to its research. Existing video dialogue models use cross-modal attention mechanisms or graph structures to capture the relevance between video semantics and dialogue context, where all visual information is treated equally in the calculation. This leads to the model easily ignoring some fine-grained spatio-temporal information, such as the continuous movement of the same object over time or the object information at non-prominent positions in the image, thereby reducing the performance of video dialogue.
[0003] The goal of the video dialogue generation task is to combine video information and dialogue context to reason and generate fluent natural language to communicate with humans based on the given scene, given video, historical dialogue, and specific questions. Similar to the former VQA task, video dialogue requires in-depth understanding of visual concepts and relationships in the video and reasoning and answering based on the current question. Compared with visual question answering, video dialogue needs to deal with a large amount of spatio-temporal visual information and multi-modal relationships, facing unique challenges: 1) using a global approach to process video information, ignoring the unique temporal and spatial features of the video, treating all visual information equally, resulting in the inability to effectively capture fine-grained spatio-temporal information; 2) the scenes involved in video dialogue are complex and diverse, and the dialogue quality and effect stability of existing video dialogue methods differ greatly in different scenes; 3) video dialogue needs to deeply understand and express reasoning of multi-modal information from multiple levels such as syntax, semantics, and logic, with high requirements for the coherence and naturalness of generated content.
[0004] To address the above problems, related art research uses recurrent neural networks to encode dialogue history, and later methods use attention mechanisms. However, all visual information in the video is treated equally in the calculation, ignoring the unique temporal and spatial features of the video, which leads to the model easily ignoring some fine-grained important information. SUMMARY
[0005] In order to avoid the shortcomings of the prior art, the present application provides a rich semantic video dialogue generation method based on hierarchical visual attention, to solve the problem that the prior art ignores the unique temporal and spatial features of the video, thereby causing the model to easily ignore some fine-grained important information.
[0006] According to an embodiment of the present disclosure, a rich semantic video dialogue generation method based on hierarchical visual attention is provided, which comprises:
[0007] segmenting an input video into a plurality of video clips;
[0008] extracting picture features and motion features of the video clips by using a C3D-rgb model and a C3D-flow model respectively, and extracting multi-dimensional audio features of the video clips by using a VGG model;
[0009] processing the picture features, the motion features and the multi-dimensional audio features to obtain video-audio features;
[0010] The attention memory unit analyzes the words of the input question one by one to obtain word embedding vectors.
[0011] The attention memory unit performs fine adjustment on the word embedding vectors, the input question and the video-audio features at each time step to generate a dialogue reply to the input question.
[0012] The attention memory unit comprises an attention module, a channel fusion module, a memory module, an optimization module and a plurality of Transformer operation blocks.
[0013] In the step of analyzing the words of the input question one by one by the attention memory unit to obtain word embedding vectors, the step comprises:
[0014] The embedding layer in the attention memory unit is used to convert the words of the input question into corresponding word embedding vectors; wherein the word embedding vectors contain semantic information of the current words.
[0015] After the step of analyzing the words of the input question one by one by the attention memory unit to obtain word embedding vectors, the step further comprises:
[0016] The word embedding vectors are sent to the memory module to enable the memory module to remember information of the input question.
[0017] In the step of performing fine adjustment on the word embedding vectors, the input question and the video-audio features at each time step by the attention memory unit to generate a dialogue reply to the input question, the step comprises:
[0018] The attention module performs an initial attention mechanism on all the video-audio features based on the current word embedding vectors, and with respect to the video-audio features associated with the current words.
[0019] taking a weighted sum of the picture feature and the motion feature in the video-audio feature, and fusing with the channel fusion module to obtain an intermediate fusion representation;
[0020] adding the hidden layer state of the memory module, the previous video representation and the intermediate fusion representation as the input of the memory module to enable the memory of all executed attention operations;
[0021] The attention module uses the hidden layer state of the memory module to perform a second attention mechanism on all the video-audio features;
[0022] refining the first attention weight of the initial attention mechanism and the second attention weight of the second attention mechanism in the optimization module to generate the current video representation to obtain the dialogue reply.
[0023] In the step of dividing the input video into a plurality of video segments, the step comprises:
[0024] For the input video, the input video is first divided into a plurality of video segments, and each video segment contains N video frames in a sliding window.
[0025] The table of the video-audio feature is shown as:
[0026] VA t =[V rgb , V flow , A vgg ]
[0027] Wherein, VA represents a video-audio feature, t represents the tth video-audio feature, V rgb represents a picture feature, V flow represents a motion feature, and A vgg represents a multi-dimensional audio feature.
[0028] The technical scheme provided by the embodiments of the present disclosure can include the following beneficial effects:
[0029] In the embodiment of the present disclosure, by the above-mentioned rich semantic video dialogue generation method based on hierarchical visual attention, on the one hand, for a given input video, firstly, picture features and motion features of the input video are extracted, then the proposed input question is analyzed word by word, and these features are finely adjusted by an attention memory unit in each time step. After processing the last word of the input question, the most relevant and valuable visual attention is generated to answer the specific question and give the answer. At the same time, attention is also paid to the historical dialogue context as a dialogue generation reference, fully integrating picture and motion features to obtain a dialogue reply generated for the input question. On the other hand, information is extracted from the picture features and motion features of the video, and its attention mechanism is gradually improved to consider the coarse-grained features and fine-grained word features of the question; it has strong scalability and can be applied to information channels containing text, audio and other modalities in the future; at the same time, it can improve efficiency and accuracy by sharing parameters and learning the correlation between multiple tasks. BRIEF DESCRIPTION OF DRAWINGS
[0030] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.
[0031] Figure 1 A step diagram of a rich semantic video dialogue generation method based on hierarchical visual attention in an exemplary embodiment of the present disclosure is shown;
[0032] Figure 2 A video dialogue generation model architecture diagram in an exemplary embodiment of the present disclosure is shown;
[0033] Figure 3 An example diagram of extracting picture features and motion features based on hierarchical visual attention in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0034] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations, however, can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.
[0035] In addition, the accompanying drawings are only schematic and are non-limiting illustrative of the embodiments of this disclosure; they are not necessarily drawn to scale. Same reference numerals in different drawings represent the same or similar functionalities unless otherwise indicated. Some of the drawings can be schematic or sketch-like representations of processes, devices, etc., and are not necessarily drawn to scale.
[0036] In the present example embodiment, a rich semantic video dialogue generation method based on hierarchical visual attention is first provided. Referring to FIG. 1, the rich semantic video dialogue generation method based on hierarchical visual attention can include steps S101-S105. Figure 1
[0037] Step S101: dividing an input video into a plurality of video segments;
[0038] Step S102: extracting picture features and motion features of the video segments by using C3D-rgb model and C3D-flow model respectively, and extracting multi-dimensional audio features of the video segments by using VGG model;
[0039] Step S103: processing the picture features, the motion features and the multi-dimensional audio features to obtain video-audio features;
[0040] Step S104: analyzing words of an input question one by one by an attention memory unit to obtain word embedding vectors;
[0041] Step S105: refining the word embedding vectors, the input question and the video-audio features by the attention memory unit in each time step to generate a dialogue reply to the input question.
[0042] Through the above rich semantic video dialogue generation method based on hierarchical visual attention, on the one hand, for a given input video, picture features and motion features of the input video are first extracted, then the proposed input question is analyzed word by word, and these features are refined by the attention memory unit in each time step. After processing the last word of the input question, the most relevant and valuable visual attention is generated to answer the specific question and give the answer. At the same time, the historical dialogue context is also paid attention to as a dialogue generation reference, and the picture and motion features are fully integrated to obtain the dialogue reply to the input question. On the other hand, information is extracted from the picture features and motion features of the video, and the attention mechanism is gradually improved to consider the coarse-grained features of the question and the fine-grained word features; the scalability is strong, and it can be applied to information channels containing text, audio and other modalities in the future; at the same time, the correlation between multiple tasks can be learned by sharing parameters to improve efficiency and accuracy.
[0043] In the following, reference will be made toFigures 1 to 3 The steps of the semantically rich video dialogue generation method based on hierarchical visual attention described in this example embodiment will be explained in more detail.
[0044] In step S101, for a given video V k First, V needs to be k Divided into T k There are N video segments, and each video segment contains N video frames f in a sliding window.
[0045] In steps S102 and S103, for each video segment S t ={f1, f2, ..., f l Both C3D-rgb and C3D-flow models are used to extract multi-dimensional video feature information, i.e., image features V. rgb and motion characteristics V flow Due to the synchronization between the corresponding audio and video data, the VGG model can be used to select audio from the same video segment to extract its multidimensional audio features. After the above processing, the obtained features are concatenated to obtain the video-audio feature VA.
[0046] VA t =[V rgb V flow A vgg ]
[0047] The obtained video-audio features (VA) are fed into a fully connected layer and mapped to a word embedding space with the same dimension as the text features. To fully utilize the sequential nature of the sequence and enable the model to effectively distinguish different input parts, including video information, video captioning, and users, the final representation of each word tag is obtained by adding its word embedding, positional encoding, and segment embedding.
[0048] like Figure 2 The image shown is an example of extracting image features and motion features based on hierarchical visual attention.
[0049] In steps S104 and S105, this application applies a video dialogue generation model during the sequential processing of the question words, the architecture of which is as follows: Figure 3 As shown, the model first uses an embedding layer to convert the input words into their word embedding vectors x. t This contains the semantic information of the current word. Then, x... t Feed into the memory module LSTM qThe hidden state of this layer is considered to have remembered information about the problem being addressed. At this point, the word embedding vector x... t and LSTM q The hidden layer states are simultaneously input into the Attention Memory Unit (AMU) to generate more refined visual attention hierarchically for video image features and motion features.
[0050] The AMU first takes the current word embeddings, question information, and video features as input, and then progressively refines the video feature attention. The AMU mainly includes four operational blocks: Attention Module (ATT), Channel Fusion Module (CF), and Memory Module (LSTM). a The four main module operations, along with several Transformer operation blocks, constitute the attention memory unit.
[0051] The execution process of each operation block in AMU is as follows: First, ATT1 is based on the current word embedding x t An initial attention mechanism is applied to F, focusing on video features associated with the current word. Then, the video frame features V are... rgb With motion characteristics V flow The weighted sum is then fused with the cross-validation (CF) to assign a weight score to each channel, resulting in the intermediate fused representation of the video, u. t At this point, LSTM... q Hidden layer state The previously generated video represents v t-1 and the intermediate video indicates u t The sum of the three constitutes an LSTM. a The input allows it to memorize all the attention operations performed. Then, ATT2 uses... A second attention mechanism is performed on F, refining the initial attention weights in REF. and first attention weight Generate video representation v t It can be used in the next time step.
[0052] By incorporating visual and linguistic attention mechanisms into the dialogue generation process, joint modeling of video and language is achieved. Different levels of local attention are allocated to different regions of the video based on the user's questions, while incorporating historical dialogue context information, thereby generating semantically richer and more coherent dialogue content.
[0053] It can be seen that, by modeling hierarchical attention, the application first captures global visual semantic information using global visual attention according to the dialogue context, and locates the video time sequence / spatial range of the dialogue input attention, and then further captures fine-grained visual information using a local attention mechanism, and generates a corresponding reply by combining a multi-task learning method. Through this method, the model further captures fine-grained visual information using a local attention mechanism, obtains the spatio-temporal correspondence between the multi-modal data pairs, realizes cross-modal alignment between videos and texts at different times, and thus generates more effective high-quality replies.
[0054] Further description will be made below in combination with simulation tests.
[0055] 1. Experimental data
[0056] In order to better explore the problem of audio-visual scene understanding and spatio-temporal reasoning, the data set DSTC7 AVSD is adopted in this paper. Compared with the AVSD data set, this data set has the characteristics of large data scale, focus on common sense problems, wide coverage of dialogue content and high quality. Table 1 shows the relevant statistical data of the DSTC7 AVSD data set.
[0057] Table 1 Statistics for the DSTC7 AVSD dataset
[0058] Table 1 Statistics for the DSTC7 AVSD dataset
[0059]
[0060] The data set has the characteristics of multi-round, multi-user, multi-level text information and diverse dialogue style. Each video scene contains multiple rounds of dialogue, involves multiple speakers, and contains multiple levels of information such as video description, question, answer, etc., and the topic range is wide. In addition, the video and audio data in this data set are collected in real scenes, covering various noises in natural environments, so that the model can have stronger generalization ability.
[0061] First, the pre-training weights of the model are initialized based on the GPT3 model. In the pre-training process, three rounds of historical dialogue information are used, the size of the model hidden layer is set to 768, the batch size is set to 4, and the Adam optimizer with a learning rate of 6.25e-5 is used. In the decoding process, the beam search algorithm (Beam Search) is mainly used, the beam size is set to 5, the maximum length is set to 20, and the length penalty term is set to 0.3.
[0062] 2. Comparative method analysis
[0063] This section mainly selects four related methods, Naive Fusion, MTN, BiST and STSGR, and compares their experimental results with the model.
[0064] (1) Naive Fusion
[0065] Naive Fusion is a simple model for multi-turn dialogue systems, which mainly combines modal information with projection matrices, concatenates historical dialogue text and inputs it into a single neural network model for processing to generate dialogue.
[0066] (2) MTN
[0067] MTN is a neural network model based on multi-head attention mechanism, which can generate good dialogue responses in a multi-modal setting, mainly including encoder, decoder and auto-encoder layers.
[0068] (3) BiST
[0069] BiST is a visual language neural framework based on text clues for high-resolution video queries. Through bidirectional reasoning in time and space, it learns dynamic information diffusion between two feature spaces.
[0070] (4) STSGR
[0071] STSGR is a multi-modal substitution reasoning framework based on semantic control, composed of a series of Transformer modules. Each module accepts one modality as input and generates representations based on the input question.
[0072] 3. Experimental results
[0073] For the video dialogue generation task, the commonly used evaluation metrics mainly include manual evaluation and automatic evaluation. Since manual evaluation requires a lot of time and cost, and has the problems of subjectivity and irreproducibility, this paper adopts automatic evaluation metrics to measure the quality of the generated dialogue response. Common automatic evaluation metrics include BLEU, ROUGE, METEOR, CIDEr, etc.
[0074] Using the above evaluation metrics, the experimental results of the model and the baseline methods are shown in Table 2.
[0075] Table 2 Results of experiments
[0076]
[0077] The experimental results show that compared with the previous SOTA model BiST for this task, the model designed in this paper has achieved great improvement. Among them, the improvement of the METEOR index is the most significant, which increases from 0.284 to 0.350, an increase of 23.24%. Compared with the pure text task NF and the STSGR task, the performance gain obtained by the model in this paper is more obvious. This shows that the hierarchical visual attention mechanism video understanding method can be effectively used for such tasks. Compared with MTN, the BLEU-4 index is slightly lower, but the overall performance has been greatly improved.
[0078] Through the above rich semantic video dialogue generation method based on hierarchical visual attention, on the one hand, for a given input video, first extract the picture features and motion features of the input video, then analyze the proposed input question word by word, and in each time step, the attention memory unit is used to fine-tune these features. After processing the last word of the input question, the most relevant and valuable visual attention is generated to answer the specific question and give the answer. At the same time, attention will also be paid to the historical dialogue context as a dialogue generation reference, fully integrating picture and motion features to obtain a dialogue reply to the input question. On the other hand, information is extracted from the picture features and motion features of the video, and the attention mechanism is improved step by step to consider the coarse-grained features and fine-grained word features of the question; It has strong scalability and can be applied to information channels containing text, audio and other modalities in the future; At the same time, it can improve efficiency and accuracy by sharing parameters and learning the correlation between multiple tasks.
[0079] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.
[0080] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known or customary practices in the art to which the present disclosure pertains. The specification and examples are only considered to be exemplary, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A method for rich semantic video dialogue generation based on hierarchical visual attention, characterized in that, The method comprises: segmenting an input video into a plurality of video clips; extracting picture features and motion features of the video clips using C3D-rgb model and C3D-flow model respectively, and extracting multi-dimensional audio features of the video clips using VGG model; processing the picture features, the motion features and the multi-dimensional audio features to obtain video-audio features; The attention memory unit analyzes the words of the input question one by one to obtain word embedding vectors; wherein the attention memory unit comprises an attention module, a channel fusion module, a memory module, an optimization module and a plurality of Transformer operation blocks; The word embedding vectors are sent to the memory module to enable the memory module to remember the information of the input question; In each time step, the word embedding vectors, the input question and the video-audio features are refined by the attention memory unit to generate a dialogue reply to the input question; specifically including: The attention memory unit performs an initial attention mechanism on all the video-audio features based on the current word embedding vectors, and the video-audio features associated with the current word; The picture features and the motion features in the video-audio features are taken as a weighted sum and fused with the channel fusion module to obtain an intermediate fusion representation; The hidden layer state of the memory module, the previous video representation and the intermediate fusion representation are added as the input of the memory module to enable the memory of all executed attention operations; The attention module uses the hidden layer state of the memory module to perform a second attention mechanism on all the video-audio features; In the optimization module, the first attention weight of the initial attention mechanism and the second attention weight of the second attention mechanism are refined to generate the current video representation to obtain the dialogue reply; wherein the current video representation is used in the next time step.
2. The rich semantic video dialogue generation method based on hierarchical visual attention according to claim 1, characterized in that, In the step of analyzing the words of the input question one by one to obtain word embedding vectors, it comprises: Using the embedding layer in the attention memory unit to convert the words of the input question into corresponding word embedding vectors; wherein the word embedding vectors contain the semantic information of the current word.
3. The rich semantic video dialogue generation method based on hierarchical visual attention according to claim 1, characterized in that, In the step of segmenting the input video into a plurality of video clips, it comprises: For the input video, first segment the input video into a plurality of video clips, and each sliding window of the video clip contains N video frames.
4. The rich semantic video dialogue generation method based on hierarchical visual attention according to claim 1, characterized in that, The table of the video-audio features is shown as: wherein, VA denotes a video-audio feature, t denotes the t-th video-audio feature, V rgb denotes a picture feature, V flow denotes a motion feature, A vgg denotes a multi-dimensional audio feature.