A multi-modal hierarchical video description generation method and system

Abstract

The application discloses a multi-modal hierarchical video description generation method and system, including a candidate event extraction and description generation process and a multi-modal hierarchical paragraph description generation process, improves the video paragraph description generation task, generates a complete and more accurate paragraph-level description for a video containing multiple events, alleviates the problem of insufficient event detection accuracy of the original model, strengthens the utilization of multi-modal multi-level information of the description model, and improves the overall quality of the video description. It belongs to the cross technical field of artificial intelligence and natural language processing.

Description

Technical Field

[0001] This invention provides an automatic video description generation technology, specifically involving a multimodal hierarchical video description generation method and system, belonging to the interdisciplinary field of artificial intelligence and natural language processing. Background Technology

[0002] The purpose of the video paragraph captioning task is to generate a complete paragraph-level description for a video containing multiple events. The video paragraph captioning frameworks described in references [1] and [2] can be divided into two main modules: an event detection module and a paragraph description module. The event detection module predicts video segments containing events from the original video, and the paragraph description module generates and integrates descriptions for these detected events. Existing video paragraph captioning frameworks are limited by the low accuracy of event detection, resulting in limited paragraph description generation. Reference [3] proposes a new framework based on candidate events, which includes three modules: candidate event extraction, candidate event description, and paragraph description, which alleviates the problem of low event detection accuracy. However, the video paragraph description technology described in reference [3] mainly uses textual information to synthesize the final description, ignoring the joint use of multimodal information, resulting in low accuracy in generating video paragraph descriptions.

[0003] References

[0004] [1]MART:Memory-Augmented Recurrent Transformer for Coherent VideoParagraph Captioning.Jie Lei et.al.ACL 2020.

[0005] [2]End-to-End Dense Video Captioning with Parallel Decoding.Teng Wanget.al.ICCV 2021.

[0006] [3]Video Paragraph Captioning as a Text Summarization Task. Hui Liuet. al. ACL 2021. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, this invention provides a multimodal hierarchical video description generation method and system, which improves the video paragraph captioning task by generating a more complete and accurate paragraph-level description for a video containing multiple events, thereby solving the performance defects of existing methods in video event detection and video paragraph caption generation.

[0008] The technical solution provided by this invention is:

[0009] A multimodal hierarchical video description generation method includes the following steps:

[0010] 1) Candidate event extraction and description generation;

[0011] 11) Obtain candidate events based on the input video;

[0012] The specific process is as follows: Calculate the boundary matching score of all possible events contained in the input video, and extract the top 100 candidate events {p1,…,p} with the highest scores for each video based on this score. 100}, where each candidate event is represented by a start time and an end time;

[0013] 12) Based on the extracted candidate events, perform visual encoding and output a one-sentence description for each candidate event, i.e., the sentence-level description corresponding to the candidate event;

[0014] Specifically, a visual Transformer model is used for each event p. i Generate sentence-level natural language descriptions c i The visual Transformer model here is trained on a video description dataset. It takes candidate events as input, performs visual encoding, and outputs a sentence describing the corresponding candidate event.

[0015] 2) Multimodal hierarchical paragraph description generation, including multimodal hierarchical encoding and decoding;

[0016] 21) A multimodal hierarchical encoder is used to hierarchically encode multimodal (visual-text) information to obtain the feature representation of candidate events and the feature representation of the sentence-level descriptions corresponding to the candidate events;

[0017] Video frames and events belong to the visual modality, while sentence descriptions and words belong to the text modality. Multimodal hierarchical encoders include visual encoders and text encoders. Both visual and text encoders consist of a self-attention layer, a cross-modal attention layer, a feedforward neural network layer, and an attention pooling layer. Fine-grained feature representations are obtained through the first three layers, and then coarse-grained feature representations are obtained through the pooling layer. The hierarchical encoding process includes:

[0018] 211) Based on the candidate events and their corresponding sentence-level descriptions, obtain video frame features and word features;

[0019] 212) The video frame features are fed into the visual encoder, and the context information is fused through the self-attention layer. The cross-modal information fusion is achieved by interacting with the intermediate representation of the text encoder (i.e. the output features of the self-attention layer) through the cross-modal attention layer. Then, the video frame features after fusing multimodal information are obtained through the feedforward neural network layer.

[0020] 213) The word features are fed into the text encoder, and contextual information is fused through the self-attention layer. Cross-modal information fusion is achieved by interacting with the intermediate representation of the visual encoder (the output features of the self-attention layer) through the cross-modal attention layer. The word features that have fused multimodal information are obtained through the feedforward neural network layer.

[0021] The visual encoder and text encoder are symmetrical, both using the output of another encoder's self-attention layer for cross-modal interaction.

[0022] 214) After obtaining the final video frame features and word features, the attention pooling layer of the visual encoder is used to synthesize the video frame features of the candidate events to obtain the feature representation of the candidate events. The attention pooling layer of the text encoder is used to synthesize the word features to obtain the feature representation of the corresponding sentence-level description.

[0023] That is, fine-grained video frame-word features and coarse-grained candidate event-sentence features are obtained through encoder encoding.

[0024] 3) Decoding is performed through a multimodal hierarchical decoder to generate paragraph-level descriptions; the multimodal hierarchical decoder includes a word embedding layer, a positional encoding layer, a masked self-attention layer, a candidate event-sentence level attention layer, a video frame-word level attention layer, a gating unit, a feedforward neural network, and a softmax function module;

[0025] The decoding process includes:

[0026] 31) First, the generated descriptive text is encoded through a word embedding layer and a positional encoding layer to obtain the encoded text; during the first step of text generation, the start marker is... The encoding and decoder generate the first word w1, and then the second step is... The word w1 is encoded to generate the second word w2, and this process continues until the end marker is generated. <e>Or reach the specified length.

[0027] 32) The text to be encoded is fed into a masked self-attention layer; coarse-grained features are then focused on through a candidate event-sentence level attention layer, and fine-grained features are then focused on through a video frame-word level attention layer.

[0028] 33) The proportion of visual and text features is controlled by gating units. Specifically, the mask is fed from the attention layer output into the sigmoid function to calculate the proportion s between [0,1]. Then, s and 1-s are used to perform a weighted summation of visual and text features.

[0029] 34) Text prediction and generation are performed using a feedforward neural network and a softmax function to generate paragraph-level descriptions;

[0030] The above steps enable the generation of multimodal hierarchical video descriptions.

[0031] The present invention also provides a system employing the above-mentioned multimodal hierarchical video description generation method, comprising: a candidate event extraction and description generation module, and a multimodal hierarchical segment description generation module.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] The multimodal hierarchical video description generation method and system provided by this invention improves the video paragraph captioning task, generating a more complete and accurate paragraph-level description for a video containing multiple events. This alleviates the problem of insufficient event detection accuracy in the original model, enhances the utilization of multimodal and multi-level information by the description model, and improves the overall quality of video description. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention.

[0035] Figure 2 This is a structural diagram of the multimodal hierarchical encoder used in the paragraph description module of this invention.

[0036] Figure 3 This is a structural diagram of the multimodal hierarchical decoder used in the paragraph description module of this invention.

[0037] Figure 4 This is an example of video description generation according to the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0039] This invention provides a multimodal hierarchical video description generation method and system, which improves the video paragraph captioning task and generates a complete and more accurate paragraph-level description for a video containing multiple events.

[0040] For convenience, the notation used in the method of this invention includes: input video denoted as V, and final output segment-level description denoted as P. A video may contain multiple candidate events {p}. i }, a candidate event p i Refers to a collection of n consecutive video frames (f0,...,f...). n (The video clip is shown.) For each candidate event, we first generate a sentence-level description c containing m words. i =(x0,…,x m Candidate events and their sentence-level descriptions {(p i ,c i The paragraph description module generates a paragraph-level description P.

[0041] The system framework and method flow of this invention are as follows: Figure 1 As shown, it mainly includes two steps: candidate event extraction and description generation, and multimodal hierarchical paragraph description generation. The specific steps are as follows:

[0042] A. Candidate event extraction and description generation. The implementation method is as follows:

[0043] A1. The system receives video input V and uses the BMN (Boundary-Matching Network) model (reference: BMN: Boundary-Matching Network for Temporal Action Proposal Generation. Tianwei Lin et al. ICCV 2019) to calculate the boundary matching score of all possible events. Based on this score, the system extracts the top 100 candidate events {p1,…,p} with the highest scores for each video. 100 }, where each candidate event is represented by a start time and an end time;

[0044] A2. Based on the extracted candidate events, a visual Transformer model is used for each event p. i Generate sentence-level natural language descriptions c i The visual Transformer model here is trained on a video description dataset (specifically, the ActivityNet Captions dataset and the YouCookII dataset were used in the implementation). It takes candidate events as input, performs visual encoding, and outputs a one-sentence description of the corresponding candidate event.

[0045] B. Multimodal hierarchical paragraph description generation. The implementation method is as follows:

[0046] B1. Multimodal hierarchical coding. After step A, for each input video V, we obtain 100 candidate events and their corresponding sentence-level descriptions {(p i ,c i We use the CLIP model (reference: Learning Transferable Visual Models From Natural Language Supervision. A Radfordet et al.) to extract video frame features. and word features Where n is the number of candidate events, n i m is the number of frames contained in the i-th candidate event. i This represents the number of words contained in the description of the i-th candidate event. Video frame features and word features are fed into a multimodal hierarchical encoder to further encode visual and textual modal information. The structure of the multimodal hierarchical encoder is as follows: Figure 2 As shown, it includes a visual encoder and a text encoder, which perform hierarchical encoding of candidate events and their corresponding sentence-level descriptions.

[0047] Video frame features are fed into the visual encoder ( Figure 2 The upper part (of the structure) contains a self-attention layer, a cross-modal attention layer, a feedforward neural network layer, and an attention pooling layer. Video frame features are fused with contextual information through the self-attention layer and then combined with the text encoder through the cross-modal attention layer. Figure 2 The self-attention layer (in the lower half) outputs features to achieve cross-modal information fusion, and then obtains video frame features that fuse multimodal information through a feedforward neural network layer. Then, the attention pooling layer synthesizes the fine-grained video frame features to obtain a coarse-grained candidate event feature representation {p1,...,p...}. n Similarly, word features are fed into the text encoder (which, like the visual encoder, includes a self-attention layer, a cross-modal attention layer, and a feedforward neural network layer). Through these layers, word features incorporating multimodal information are obtained. Then, by fusing fine-grained word features through an attention pooling layer, a coarse-grained sentence feature representation {c1,...,c} is obtained. n }

[0048] B2. Multimodal Hierarchical Decoding. Following step B1, the encoder obtains fine-grained video frame-word features and coarse-grained candidate event-sentence features. We use a multimodal hierarchical decoder to generate paragraph-level descriptions based on these features. The decoder structure is as follows: Figure 3 As shown, it includes a word vector embedding layer, a position encoding layer, a masked self-attention layer, a candidate event-sentence level attention layer, a video frame-word level attention layer, a gating unit, a feedforward neural network, and a softmax function module.

[0049] The multimodal hierarchical decoder generates paragraph-level descriptions P word by word through an autoregressive generation method. The generated description portion (starting with the input marker at the beginning of generation) is already in progress. First, words are encoded using a word embedding layer and a positional encoding layer. Then, the encoded words are fed into a masked self-attention layer to fuse with contextual information. To better utilize the multimodal and multi-level feature representations obtained from the encoder, the decoder sequentially uses a candidate event-sentence level attention layer to focus on coarse-grained features and a video frame-word level attention layer to focus on fine-grained features. After these two attention layers, we obtain visual modal features (hidden representations after focusing on candidate event features and video frame features) and text modal features (hidden representations after focusing on sentence features and word features). Next, a gating unit performs weighted fusion of the visual and text modal features, and finally, text prediction is generated using a feedforward neural network and a softmax function.

[0050] During model training (simultaneously training the encoder and decoder), we used a cross-entropy loss function and an attention loss function for supervised training. The cross-entropy loss function trained the model to generate the target paragraph description P, while the attention loss function trained the model to focus on candidate events and sentence-level descriptions related to the currently generated text during decoding (using the overlap ratio IoU as the relevance of candidate events and text similarity as the relevance of sentence-level descriptions).

[0051] The multimodal hierarchical video description generation method and system provided by this invention alleviate the problem of insufficient event detection accuracy in the original model, enhance the utilization of multimodal and multi-level information by the description model, and improve the overall quality of video description.

[0052] In specific implementation, the model settings of this invention are as follows: The CLIP model is used to extract the initial video frame features and word features; the BMN model is used to extract the 100 highest-scoring candidate events for each video; and the visual Transformer model is used for candidate event description. In the paragraph description generation module, each candidate event is sampled at equal intervals for a maximum of 80 frames. The visual encoder has a hidden layer size of 384 and 8 attention heads; the text encoder has a hidden layer size of 512 and 8 attention heads; the encoder has 1 layer; and the decoder has 2 layers. The model was trained using the ActivityNet Captions dataset and the YouCookII dataset. The batch size for the ActivityNet Captions dataset was set to 36, and the batch size for the YouCookII dataset was set to 12. The RAdam optimizer was used to optimize the neural network, with a learning rate of 3.6 × 10⁻⁶. -4 The model was trained on a single 1080Ti graphics card. The training time on the ActivityNetCaptions dataset was about 10 hours, and the training time on the YouCookII dataset was about 5 hours. Finally, our system achieved a Bleu@4 score of 11.15 on the ActivityNetCaptions dataset and a Bleu@4 score of 7.47 on the YouCookII dataset, which surpassed the video description performance of existing methods [3]. Figure 4 An example of video description generation is provided, in which the video descriptions obtained by using manual description, the method described in reference [1] and the method of this invention are respectively:

[0053] Artificial description: A group of children are seen swimming in a pool.The

[0054] kids hit a ball back and forth in the water. They fight over the ball,

[0055] trying to get into the goal.

[0056] Method [1]: A large group of people are seen swimming around a beach

[0057] withakite.Severalclips are shown of people playing water polo and

[0058] throwing the ball around . More people are shown swimming around

[0059] the pool while the camera captures their movements.

[0060] This method: A group of people are in a swimming pool.They are

[0061] engaged in a game of water polo.They swimand throw the ballback

[0062] and forth.

[0063] Compared with the generation results of the method described in reference [1], the method in this invention can generate more accurate and concise paragraph descriptions, better describe the process of the entire video, and is closer to human description.

[0064] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims. < / e>

Claims

1. A multimodal hierarchical video description generation method, characterized in that, Includes the following steps: 1) Based on the input video, extract candidate events from the video and generate sentence-level descriptions of the corresponding candidate events; 2) Perform multimodal hierarchical coding, i.e., obtain fine-grained video frame-word features and coarse-grained candidate event-sentence features through encoder encoding; including: 21) A multimodal hierarchical encoder is used to hierarchically encode the multimodal information to obtain the feature representation of the candidate event and the feature representation of the sentence-level description corresponding to the candidate event; the multimodal information includes visual modality and text modality; The visual modality includes video frames and events; the text modality includes sentence descriptions and words; the multimodal hierarchical encoder includes a visual encoder and a text encoder; both the visual encoder and the text encoder have structures including a self-attention layer, a cross-modal attention layer, a feedforward neural network layer, and an attention pooling layer; fine-grained feature representations are obtained through the first three layers, and then coarse-grained feature representations are obtained through the pooling layer; the multimodal hierarchical encoding process includes: 211) Based on the candidate events and their corresponding sentence-level descriptions, obtain video frame features and word features; 212) The video frame features are fed into the visual encoder, and the context information is fused through the self-attention layer. The cross-modal information fusion is achieved by interacting with the output features of the self-attention layer of the text encoder through the cross-modal attention layer. The video frame features after fusing multimodal information are then obtained through the feedforward neural network layer. 213) The word features are fed into the text encoder, and the context information is fused through the self-attention layer. The cross-modal information fusion is achieved by interacting with the output features of the self-attention layer of the visual encoder through the cross-modal attention layer. The word features fused with multimodal information are obtained through the feedforward neural network layer. 214) After obtaining the video frame features and word features fused with multimodal information, the attention pooling layer of the visual encoder is used to synthesize the video frame features fused with multimodal information of the candidate events to obtain the feature representation of the candidate events; the attention pooling layer of the text encoder is used to synthesize the word features fused with multimodal information to obtain the feature representation of the corresponding sentence-level description. 3) Decoding is performed using a multimodal hierarchical decoder to generate paragraph-level descriptions. The multimodal hierarchical decoder includes a word embedding layer, a positional encoding layer, a masked self-attention layer, a candidate event-sentence level attention layer, a video frame-word level attention layer, a gating unit, a feedforward neural network, and a softmax function module. The multimodal hierarchical decoding process includes: 31) The generated descriptive text is encoded through a word embedding layer and a positional encoding layer to obtain the encoded text; 32) The text to be encoded is fed into a masked self-attention layer; coarse-grained features are then focused on through a candidate event-sentence level attention layer, and fine-grained features are then focused on through a video frame-word level attention layer. 33) The proportion of visual-text features is adjusted through gating units; 34) Text prediction is performed using a feedforward neural network and a softmax function to generate paragraph-level descriptions; By following the steps described above, multimodal hierarchical video descriptions can be generated.

2. The method as described in claim 1, characterized in that, Step 1) further includes the following process: 11) Obtain candidate events based on the input video; the specific process is as follows: Calculate the boundary matching score for all possible events contained in the input video, and extract the top 100 candidate events {p1,…,p} for each video based on this score. 100 }, where each candidate event is represented by a start time and an end time; 12) Visually encode the extracted candidate events and output a one-sentence description for each candidate event, i.e., the sentence-level description corresponding to the candidate event; Specifically: Use the visual Transformer model for each event p i Generate sentence-level natural language descriptions c i The visual Transformer model is trained on a video description dataset. It receives candidate events as input, performs visual encoding, and outputs a sentence describing the corresponding candidate event.

3. The method as described in claim 1, characterized in that, In step 2), the CLIP model is used to extract video frame features and word features. The video frame features and word features are then fed into a multimodal hierarchical encoder to further encode the visual modality and text modality information.

4. The method as described in claim 1, characterized in that, In step 3), the multimodal hierarchical decoder specifically generates paragraph-level descriptions word by word through autoregressive generation.

5. The method as described in claim 4, characterized in that, include: Enter a start marker when generating paragraph-level descriptions; The generated paragraph-level description is first encoded into words through a word embedding layer and a positional encoding layer; The encoded words are then fed into a masked self-attention layer and fused with the preceding information. The decoder sequentially passes through the candidate event-sentence level attention layer and the video frame-word level attention layer to obtain visual modal features and text modal features; The gating unit then performs weighted fusion of visual and textual modal features; Finally, the text is predicted and generated using a feedforward neural network and a softmax function.

6. The method as described in claim 5, characterized in that, Weighted fusion specifically involves calculating the output of the mask from the attention layer using the sigmoid function to obtain the ratio s between [0,1], and then using s and 1-s to perform a weighted summation of visual and textual features.

7. The method as described in claim 1, characterized in that the model During training, the encoder and decoder are trained simultaneously. Specifically, the model is trained in a supervised manner using the cross-entropy loss function and the attention loss function. The cross-entropy loss function trains the model to generate the target paragraph description, while the attention loss function trains the model to generate candidate events and sentence-level descriptions related to the currently generated text during the decoding process.

8. A multimodal hierarchical video description generation system employing the method of claim 1, characterized in that, include: Candidate event extraction and description generation module and multimodal hierarchical paragraph description generation module.

Images