A trilinear coding system and video-language representation learning method

Abstract

This invention discloses a trilinear coding system and a method for learning video-language representations. The method comprises the following steps: S1. Using educational video and television datasets as data samples, the data samples are divided into video segments; S2. Using a dense subtitle generation model to extract dense subtitles for the video segments; extracting the original ASR transcripts; and using the dense subtitles for each segment as supplementary text to the original ASR transcripts; S3. Using the video segments, original ASR transcripts, and dense subtitles as input to the trilinear coding system to obtain a video-language representation; and S4. Using the video-language representation to perform multimodal information exchange between video and text. The method focuses on intermodal mutual information; using dense subtitles as supplementary text, the dense subtitles are automatically extracted, achieving video-text matching.

Description

Technical Field

[0001] The present invention relates to the field of multimodal representation learning, and specifically includes a trilinear coding system and a video-language representation learning method. Background Art

[0002] In the context of big data and new infrastructure, the increasing abundance of data objects makes the data itself multimodal. "Multimodality" can be intuitively understood as different multimedia forms of data, or it can be used as a more fine-grained concept. The key to whether data is multimodal lies in whether the data is heterogeneous. For example, when searching for information about a singer online, a variety of different data related to the singer will be involved, such as images, videos, text, audio, and so on. These four types of data can be understood as the multimodal data of the object. Representation learning, as one of the fundamental methods for studying multimodal data, can learn useful representations from data. This greatly reduces the reliance on feature engineering during data processing and is widely used in many specific multimodal tasks.

[0003] Leveraging the high correlation and complementarity between data from different modalities, multimodal representation learning attempts to explore the commonalities and characteristics between these modalities, eliminating modal redundancy and representing the semantics of multimodal data as real-valued vectors. These real-valued vectors embody the shared semantics and unique characteristics of the different modal data, enabling their application in processing and understanding multimodal data. In terms of the modalities processed, research in multimodal representation learning primarily focuses on image-text and video-text, or in other words, vision-language representation learning.

[0004] Currently, there are two main research frameworks for visual-language representation learning that offer good performance. One is based on contrastive learning or similarity learning, which optimizes the representation of each modality under similarity constraints. The other is based on pre-training architectures such as autoregressive or autoencoding, which use efficient neural networks like the Transformer to encode samples from various data modalities into features and then reconstruct them.

[0005] Similarity-based visual language representation learning aims to minimize the distance between different modalities within a collaborative space. Its input data primarily consists of visual-textual information pairs with ranking or positive-negative relationships. Visual language representation learning is achieved through various modeling approaches. Weston et al. used ranking loss in WSABIE to measure the similarity between annotated data and images. DeViSE, based on the concept of deep zero-shot learning, establishes a linear mapping between pre-trained vectors of different modalities. However, these similarity-based visual language representation learning models cannot be used as a unified representation output; instead, they require a concatenation method to connect the two vectors.

[0006] Visual language representation learning based on an autoregressive pre-training architecture reconstructs the details between each modal element, processing the visual and language modalities into a language sequence task. Through self-supervision, it learns a unified joint encoding of the two modalities from massive amounts of data, avoiding the need to reconstruct the encodings of different modalities required in similarity-based methods. This type of method based on an autoregressive pre-training architecture is generally divided into two stages: 1) In the first stage, the model uses autoencoding or autoregression to fully integrate the modalities of visual language information to produce high-quality visual language representations. This involves setting up a series of self-supervised learning pre-tasks and pre-training the model on a large number of pre-training datasets, so that the learned visual language joint encoding has general knowledge that is independent of the task; 2) In the second stage, the model fine-tunes the parameters of downstream tasks to make it suitable for specific multimodal tasks.

[0007] Currently, the mainstream visual language representation learning models have three main encoding methods for the input text feature vectors and visual feature vectors. The first is an encoding method based on a shared single-stream structure, which concatenates text features and visual features, and then connects a BERT-like automatic encoder for modal fusion. Unicoder-VL, VL-BERT, VideoAsMT, VideoBERT, ActBERT and UNITER all use this encoding method; the second is an encoding method based on a cross-dual-stream structure, which first uses two separate encoders to encode text features and visual features respectively, and then uses two cross-modal encoders to cross-fuse the visual and language modal encodings. Typical examples include ViLBERT and LXMERT; the third is an encoding method based on a joint dual-stream structure, which also first uses two separate encoders to encode text features and visual features respectively, and then uses a shared cross-modal encoder to fuse different modal features. CBT and HERO use this encoding method. Compared to encoding methods based on a shared single-stream architecture, the dual-stream architecture can adapt to the different processing requirements of each mode and interact at different representation depths. Compared to the cross-architecture, the joint architecture has a shared cross-modal encoder, which can achieve more comprehensive information exchange between different modalities.

[0008] For visual objects, dynamic videos contain more modal features than static images, such as vision, motion, audio, and text, which can provide richer modal information for multimodal representation learning and model stronger spatiotemporal dependencies. This has made video-language representation learning a key research topic in the field of multimodal representation learning in recent years. VideoBERT is the first to extend the BERT model to learn the joint representation of video and text. It leverages the characteristics of the BERT network autoencoder and uses the masked language model (MLM) and next sentence prediction (NSP) as pre-training tasks to fully integrate video and text features. However, VideoBERT only uses discrete tags to represent video frames, without fully utilizing other rich features of video frames. In addition, the model takes a simple concatenation of video subtitles and visual frames as input, losing the temporal alignment of video and text, which weakens the model's ability to model temporal dependencies. To address this issue, the HERO model uses a hierarchical structure to learn finer-grained alignments between video and text. It also adds a temporal Transformer to preserve the temporal correspondence between video and text. This layered structural model enables multi-granular video-text fusion. This improves the model's temporal modeling capabilities to a certain extent. However, neither the direct use of the BERT structure for concatenated modal fusion nor the use of a hierarchical structure for multi-granular modal fusion achieves sufficient interaction and fusion of both intra-modal and inter-modal information.

[0009] Furthermore, existing work on video-to-language representation learning typically trains on large-scale instructional video datasets, such as HowTo100M. This dataset contains 1.2 million videos, including 136 million video clips automatically sourced from YouTube. Each clip is paired with text transcribed from the video narrative using an automatic speech recognition (ASR) system. However, this dataset suffers from two limitations. First, the dataset lacks richness. The videos collected are limited in variety and all focus on instructional content. The scenes involved are relatively limited, and the plots are monotonous, unlike the rich scenes and interactions of people in daily life. This limits the upper limit of the model's learning capabilities. Second, the text used in the dataset is transcribed using an automatic speech recognition (ASR) system, which can suffer from semantic and temporal misalignment when compared to the visual information. Semantic misalignment specifically refers to the fact that descriptive words are sometimes irrelevant to the visual content, or that important visual objects and actions are not described in the text. Temporal misalignment refers to the fact that people may talk about something before or after the visual content appears, resulting in a temporal misalignment between the transcribed text generated by speech and the video content. Approximately 50% of the video clip and ASR transcript pairs in the HowTo100M dataset exhibit these two types of misalignments, both of which complicate the video-to-text matching task. Furthermore, ASR transcripts are inherently noisy, incomplete, and lack punctuation, which limits the language modeling capabilities of systems trained on such text. Summary of the Invention

[0010] In response to the above-mentioned deficiencies in the prior art, the present invention provides a trilinear coding system and a video-language representation learning method that solve the problems in the prior art of realizing multimodal information interaction between video and text, such as semantic errors in the correspondence between the transcribed text and the visual information, and the temporal misalignment between the presentation of the transcribed text and the visual content.

[0011] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a trilinear encoding system, including a single-mode encoder, a trilinear multimodal fusion encoder and a timing converter;

[0012] A single-mode encoder for encoding sequences of video frames, ASR-transcribed text, and dense subtitles;

[0013] A trilinear multimodal fusion encoder is used to obtain video segment-level encoding based on the output of the single-mode encoder;

[0014] Temporal transformer, which is used to learn video segment-level contextual information through pre-training tasks to obtain video-language representation;

[0015] The information interaction module is used to use video-language representations for video-text multimodal information interaction.

[0016] Furthermore, the trilinear multimodal fusion encoder includes at least one layer of multimodal fusion encoding units, each of which includes 5 multi-head attention modules, 3 self-attention modules, 3 weighted sum modules, 6 summation modules and 3 feedforward networks;

[0017] The output end of the first multi-head attention module of each layer of the multimodal fusion encoding unit is connected to the input end of the first weighted sum module and the input end of the second weighted sum module; the output end of the second multi-head attention module is connected to the input end of the first weighted sum module and the input end of the third weighted sum module;

[0018] The output of the first weighted sum module is connected to the input of the fourth multi-head attention module; the output of the second weighted sum module is connected to the input of the third multi-head attention module; the output of the third weighted sum module is connected to the input of the fifth multi-head attention module;

[0019] The output of the third multi-head attention module is connected to the input of the first summing module; the output of the fourth multi-head attention module is connected to the input of the second summing module; the output of the fifth multi-head attention module is connected to the input of the third summing module;

[0020] The output end of the first summation module is connected to the input end of the first self-attention module; the output end of the first self-attention module is connected to the input end of the first feedforward network and the input end of the fourth summation module; the output end of the first feedforward network module is connected to the input end of the fourth summation module; the output end of the second summation module is connected to the input end of the second self-attention module; the output end of the second self-attention module is connected to the input end of the fifth summation module and the input end of the second feedforward network; the output end of the third summation module is connected to the input end of the third self-attention module; the output end of the third self-attention module is connected to the input end of the third feedforward network and the input end of the sixth summation module; the output end of the fourth summation module serves as the first output end of the trilinear multimodal fusion encoder; the output end of the fifth summation module serves as the second output end of the trilinear multimodal fusion encoder; the output end of the sixth summation module serves as the third output end of the trilinear multimodal fusion encoder.

[0021] A video-language representation learning method using a trilinear coding system comprises the following steps:

[0022] S1. Use teaching video and TV datasets as data samples and divide them into video segments;

[0023] S2. Use the dense subtitle generation model to extract dense subtitles for video clips; extract the original ASR transcript; and use the dense subtitles for each clip as supplementary text for the original ASR transcript.

[0024] S3, taking the video clip, the original ASR transcription text and the dense subtitle as the input of the trilinear encoding system, performs representation learning to obtain the video-language representation;

[0025] S4. Use video-language representation for video-text multimodal information interaction.

[0026] Furthermore, the specific implementation of step S3 is as follows:

[0027] S3-1, according to the formula:

[0028] ( )

[0029] ( )

[0030] Get ASR transcribed text encoding and dense subtitle encoding ;in, 、 Respectively represent l Video clips with original ASR transcripts and dense captions; Indicates encoding of the ASR transcript of the video clip; Indicates encoding of dense subtitles;

[0031] S3-2, according to the formula:

[0032] ( )

[0033] Get video frame encoding ; Among them, Transformer represents encoding the context information of the video frame, Indicates the A sequence of video frames of a video clip;

[0034] S3-3, using a trilinear multimodal fusion encoder to obtain video segment-level encoding;

[0035] S3-4. Input the video clip-level encoding into the temporal transformer, learn the contextual information at the video clip level through pre-training pre-tasks, and obtain video-language representation.

[0036] Furthermore, the specific implementation of step S3-3 is as follows:

[0037] S3-3-1. According to the formula:

[0038]

[0039] Get the ASR transcription text-video frame information interaction result ,in, Indicates the The intermediate representation of the layer is also called the hidden layer representation; u The first one represents the trilinear multimodal fusion encoder u layer; u The ASR transcript hidden layer representation and the u The dense caption hidden layer representations of the layers are and ; Represents query parameters; express The learnable weights of express The learnable weights of express The learnable weights of Indicates that the data is processed by the multi-head attention module;

[0040] S3-3-2, according to the formula:

[0041]

[0042] Get the information interaction results between dense subtitles and video frames ,in, express The learnable weights of express The learnable weights of express The learnable weights of

[0043] S3-3-3, according to the formula:

[0044] =

[0045] Get ASR transcript encoding with multimodal information ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of

[0046] S3-3-4, according to the formula:

[0047] =

[0048] Get dense subtitle encoding ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of

[0049] S3-3-5, according to the formula:

[0050] =

[0051] Get video frame encoding ;in, 、 、 The weighted sum of ; express The learnable weights of express The learnable weights of express The learnable weights of

[0052] S3-3-6, according to the formula:

[0053]

[0054]

[0055]

[0056] Get the transcribed text encoding after processing by the self-attention module , dense caption encoding after self-attention module processing , video frame sequence encoding after processing by the self-attention module ;in, Indicates that the data is processed by the self-attention module;

[0057] S3-3-7. The outputs of the three self-attention modules are processed by a feedforward network to obtain a multimodal feature encoding that includes inter-modal interaction information and intra-modal autocorrelation information.

[0058] S3-3-8, the three obtained polymorphic feature codes are recorded as the original ASR transcribed text sequence, the dense subtitle sequence and the video frame sequence;

[0059] S3-3-9. Determine whether the current number of iterations reaches the set number of multimodal fusion coding unit layers of the modal fusion encoder; if so, proceed to step S3-3-10; otherwise, use the currently obtained original ASR transcribed text sequence, dense subtitle sequence, and video frame sequence as inputs of the trilinear coding system and return to step S3-3-1;

[0060] S3-3-10. Concatenate the latest multimodal feature codes containing inter-modal interaction information and intra-modal autocorrelation information to obtain a video segment-level code.

[0061] The beneficial effects of the present invention are as follows: the present invention not only focuses on the autocorrelation relationship within the modality, but also considers the method of interactive information between modalities; dense subtitles based on the video are used as supplementary text, and by automatically extracting dense video subtitles, important objects, attributes and action details in the salient areas of the video frame are described, providing useful signals for video-text matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Flowchart of the video-language representation learning method;

[0063] Figure 2 It is the structure diagram of the trilinear coding system;

[0064] Figure 3 This is a single-layer structure diagram of a trilinear multimodal fusion encoder. DETAILED DESCRIPTION

[0065] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0066] like Figure 1 As shown, a video-language representation learning method using a trilinear encoder includes the following steps:

[0067] S1. Use teaching video and TV datasets as data samples and divide them into video segments;

[0068] S2. Use the dense subtitle generation model to extract dense subtitles for video clips; extract the original ASR transcript; and use the dense subtitles for each clip as supplementary text for the original ASR transcript.

[0069] S3, taking the video clip, the original ASR transcription text and the dense subtitle as the input of the trilinear encoding system, performing representation learning, and obtaining the video-language representation;

[0070] S4. Use video-language representation for video-text multimodal information interaction.

[0071] The specific implementation of step S3 is as follows:

[0072] S3-1, according to the formula:

[0073] ( )

[0074] ( )

[0075] Get ASR transcribed text encoding and dense subtitle encoding ;in, 、 Respectively represent l Video clips with original ASR transcripts and dense captions; Indicates encoding of the ASR transcript of the video clip; Indicates encoding of dense subtitles;

[0076] S3-2, according to the formula:

[0077] ( )

[0078] Get video frame encoding ; Among them, Transformer represents encoding the context information of the video frame, Indicates the A sequence of video frames of a video clip;

[0079] S3-3, using a trilinear multimodal fusion encoder to obtain video segment-level encoding;

[0080] S3-4. Input the video clip-level encoding into the temporal transformer, learn the contextual information at the video clip level through pre-training pre-tasks, and obtain video-language representation.

[0081] The specific implementation of step S3-3 is as follows:

[0082] S3-3-1. According to the formula:

[0083]

[0084] Get the ASR transcription text-video frame information interaction result ,in, Indicates the The intermediate representation of the layer is also called the hidden layer representation; u The first one represents the trilinear multimodal fusion encoder u layer; u The ASR transcript hidden layer representation and the u The dense caption hidden layer representations of the layers are and ; Represents query parameters; express The learnable weights of express The learnable weights of express The learnable weights of Indicates that the data is processed by the multi-head attention module;

[0085] S3-3-2, according to the formula:

[0086]

[0087] Get the information interaction results between dense subtitles and video frames ,in, express The learnable weights of express The learnable weights of express The learnable weights of

[0088] S3-3-3, according to the formula:

[0089] =

[0090] Get ASR transcript encoding with multimodal information ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of

[0091] S3-3-4, according to the formula:

[0092] =

[0093] Get dense subtitle encoding ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of

[0094] S3-3-5, according to the formula:

[0095] =

[0096] Get video frame encoding ;in, 、 、 The weighted sum of ; express The learnable weights of express The learnable weights of express The learnable weights of

[0097] S3-3-6, according to the formula:

[0098]

[0099]

[0100]

[0101] Get the transcribed text encoding after processing by the self-attention module , dense caption encoding after self-attention module processing , video frame sequence encoding after processing by the self-attention module ;in, Indicates that the data is processed by the self-attention module;

[0102] S3-3-7. The outputs of the three self-attention modules are processed by a feedforward network to obtain a multimodal feature encoding that includes inter-modal interaction information and intra-modal autocorrelation information.

[0103] S3-3-8, the three obtained polymorphic feature codes are recorded as the original ASR transcribed text sequence, the dense subtitle sequence and the video frame sequence;

[0104] S3-3-9. Determine whether the current number of iterations reaches the set number of multimodal fusion coding unit layers of the modal fusion encoder; if so, proceed to step S3-3-10; otherwise, use the currently obtained original ASR transcribed text sequence, dense subtitle sequence, and video frame sequence as inputs of the trilinear coding system and return to step S3-3-1;

[0105] S3-3-10. Concatenate the latest multimodal feature codes containing inter-modal interaction information and intra-modal autocorrelation information to obtain a video segment-level code.

[0106] like Figure 2 As shown, a trilinear coding system includes a single-mode encoder, a trilinear multimodal fusion encoder and a time series converter;

[0107] A single-mode encoder for encoding sequences of video frames, ASR-transcribed text, and dense subtitles;

[0108] A trilinear multimodal fusion encoder is used to obtain video segment-level encoding based on the output of the single-mode encoder;

[0109] Temporal transformer, which is used to learn video segment-level contextual information through pre-training tasks to obtain video-language representation;

[0110] The information interaction module is used to use video-language representations for video-text multimodal information interaction.

[0111] like Figure 3 As shown in the figure, the trilinear multimodal fusion encoder contains at least one layer of multimodal fusion encoding units, each of which includes 5 multi-head attention modules, 3 self-attention modules, 3 weighted sum modules, 6 summation modules and 3 feedforward networks;

[0112] The output end of the first multi-head attention module of each layer of the multimodal fusion encoding unit is connected to the input end of the first weighted sum module and the input end of the second weighted sum module; the output end of the second multi-head attention module is connected to the input end of the first weighted sum module and the input end of the third weighted sum module;

[0113] The output of the first weighted sum module is connected to the input of the fourth multi-head attention module; the output of the second weighted sum module is connected to the input of the third multi-head attention module; the output of the third weighted sum module is connected to the input of the fifth multi-head attention module;

[0114] The output of the third multi-head attention module is connected to the input of the first summing module; the output of the fourth multi-head attention module is connected to the input of the second summing module; the output of the fifth multi-head attention module is connected to the input of the third summing module;

[0115] The output end of the first summation module is connected to the input end of the first self-attention module; the output end of the first self-attention module is connected to the input end of the first feedforward network and the input end of the fourth summation module; the output end of the first feedforward network module is connected to the input end of the fourth summation module; the output end of the second summation module is connected to the input end of the second self-attention module; the output end of the second self-attention module is connected to the input end of the fifth summation module and the input end of the second feedforward network; the output end of the third summation module is connected to the input end of the third self-attention module; the output end of the third self-attention module is connected to the input end of the third feedforward network and the input end of the sixth summation module; the output end of the fourth summation module serves as the first output end of the trilinear multimodal fusion encoder; the output end of the fifth summation module serves as the second output end of the trilinear multimodal fusion encoder; the output end of the sixth summation module serves as the third output end of the trilinear multimodal fusion encoder;

[0116] Figure 3 In the figure, TA, Venc, and TC represent ASR transcribed text encoding, video frame sequence, and dense subtitle encoding, respectively; Square 1, Square 2, and Square 3 represent the first weighted sum module, the second weighted sum module, and the third weighted sum module; Circle 1, Circle 2, Circle 3, Circle 4, Circle 5, and Circle 6 represent the first summation module, the second summation module, the third summation module, the fourth summation module, the fifth summation module, and the sixth summation module.

[0117] In one embodiment of the present invention, the present invention is pre-trained on the HowTo100M and TV datasets through four pre-training pre-tasks, and then migrated to multiple downstream tasks for experiments. Experimental results show that the trilinear coding system can learn robust video-language representation, realize multimodal information interaction between video and text, and achieve the best performance on four datasets of three downstream tasks; among them, the self-supervised pre-task MLM for ASR transcription text randomly covers some words in the sentence and lets the model predict these words; the self-supervised pre-task MFM for video frames randomly covers a part of the frames in the video frame sequence, and the model learns its temporal relationship based on the input video frame sequence and predicts the visual features of the next few frames; the self-supervised pre-task VSM based on the semantic relationship between them realizes video-text alignment and matching; the pre-task FOM based on the temporal nature of video frames randomly selects 15% of the frames for shuffling, and records the reordering index as , reconstructing their original chronological order.

[0118] The three downstream tasks and corresponding datasets are introduced as follows:

[0119] The multimodal video captioning task conveys video information by automatically generating multiple corresponding natural language sentences. A system's performance in this task effectively verifies whether it fully exploits multimodal visual and linguistic information to aid content understanding. The learning performance of the trilinear coding system was verified on the TVC dataset. TVC is a multimodal video captioning dataset extended from TVR, containing 262K video descriptions and 108K video clips. Standard metrics include BLEU@4, METEOR, ROUGE-L, and CIDEr-D. BLEU@4 measures word-level accuracy within a sentence. METEOR combines precision and recall, using their harmonic mean as the evaluation criterion. ROUGE-L is a recall-based similarity metric that primarily measures the adequacy and fidelity of the reference translation. CIDEr-D measures the similarity between the reference translation and the candidate translation by calculating the TF-IDF vector cosine distance between them. The fine-tuned model was compared with three baseline methods: MMT, VALUE, and HERO. The MMT method, proposed by Lei et al. in "TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval. In ECCV, 2020, pp447-46," first used pre-trained networks to extract features for each of the three modalities. This method can extract features from more modalities using different network structures. The VALUE method, proposed by Li et al. in "VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation. In NeurIPS, 2021," compared a series of feature fusion methods and used the SlowFast network structure to effectively combine video and subtitle features. The HERO method proposed by Li et al. in "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training, in EMNLP, 2020, pp. 2046-2065" differs from previous methods in that it fuses local corresponding frame and text features through a cross-modal Transformer, and then fuses these local video features with a temporal Transformer. This method can better model the correspondence between different modalities while preserving the temporal information of the video itself. The comparison results of this method with the above methods are shown in Table 1:

[0120] Table 1

[0121]

[0122] Experimental results show that the trilinear encoding system outperforms the other three methods. This superior performance indicates that the visual-linguistic representation learned by the trilinear encoding system has fully exploited the multimodal information between vision and language to facilitate content understanding.

[0123] The multimodal video question answering task aims to answer natural language questions based on a given video. The answer with the highest similarity score among all options is selected as the predicted answer. The fine-tuned model was tested on two video question answering datasets: TVQA and How2QA. TVQA is a dataset collected in a multiple-choice setting for television videos. This dataset consists of 21,793 video clips and 152,545 question-answer pairs. The video clips in the How2QA dataset are sampled from the HowTo100M tutorial videos. Accuracy is used to measure the performance of the model on TVQA and How2QA. For the TVQA dataset, we compared our trilinear encoding system with the previous PAMN model, AMN, and Yang et al. The PAMN method, proposed by Kim et al. in "Progressive attention memory network for movie story question answering. In CVPR, 2019, pp. 8337-8346," employs a progressive attention mechanism for video question answering tasks. This method identifies the time interval relevant to answering the question and adaptively fuses information from different modalities by considering the contribution of each modality. Yuan et al. proposed the AMN method in "Adversarial Multimodal Network for Movie Story Question Answering. In IEEE Trans. Multim., vol. 23, pp. 1744-1756, 2021," which employs an inverse multimodal network structure to solve video question answering tasks. The method proposed by Yang et al. in "BERT Representations for Video Question Answering. In WACV, 2020, pp. 1545-1554," also referred to in the comparison method, builds a question-answering model based on the BERT architecture. This model aligns visual and linguistic semantic information from video clips to improve video question answering. Table 2 shows the results of our method compared with these methods. The results show that the trilinear encoding system outperforms all the comparison models, further demonstrating the effectiveness of our system.

[0124] Table 2

[0125]

[0126] The video and language reasoning task requires the model to infer whether a written statement contains or contradicts the content of a given video clip. This is a challenging new task because the visual and textual clues obtained from the video require in-depth understanding and inference of the video content. The performance of the trilinear coding system was verified on the VIOLIN dataset. The VIOLIN dataset has 15.9K video clips, which consist of 95.3K video hypothesis pairs. Similar to the video question-answering task, Accuracy is a measure of the performance of the model on the VIOLIN dataset. The VIOLIN dataset was proposed by Liu et al. in the paper "Violin: A Large-Scale Dataset for Video-and-Language Inference. InCVPR, 2020, pp10897-10907". The paper also proposed a Multi-stream method for solving video language reasoning tasks. Our method outperforms the recent HERO and Multi-stream methods on the video language inference task. The results are shown in Table 2. Combining the performance of the three tasks, we can see that the trilinear coding system proposed in this paper can model the interaction information between modalities and the autocorrelation relationship within the modality, fully exploiting the correspondence between vision and language, and thus learning visual-language representations.

[0127] This suggests that implementing multimodal information interaction between video and text can be used to determine whether image content matches text content, as well as automatically match text content to the image content. This invention not only considers autocorrelation within modalities but also considers methods for information interaction between modalities. By automatically extracting dense video captions as supplementary text, dense video captions are automatically extracted to describe important objects, attributes, and action details in prominent areas of video frames, providing useful signals for video-to-text matching.

Claims

1. A trilinear coding system, characterized in that: A single-mode encoder for encoding sequences of video frames, ASR-transcribed text, and dense subtitles; A trilinear multimodal fusion encoder is used to obtain video segment-level encoding based on the output of the single-mode encoder; Temporal transformer, which is used to learn contextual information of video segment-level encoding through pre-training tasks to obtain video-language representation; Information interaction module, used for video-text multimodal information interaction using video-language representation; The trilinear multimodal fusion encoder contains at least one layer of multimodal fusion encoding units, each of which includes 5 multi-head attention modules, 3 self-attention modules, 3 weighted sum modules, 6 summation modules and 3 feedforward networks; The output end of the first multi-head attention module of each layer of the multimodal fusion encoding unit is connected to the input end of the first weighted sum module and the input end of the second weighted sum module; the output end of the second multi-head attention module is connected to the input end of the first weighted sum module and the input end of the third weighted sum module; The output of the first weighted sum module is connected to the input of the fourth multi-head attention module; the output of the second weighted sum module is connected to the input of the third multi-head attention module; the output of the third weighted sum module is connected to the input of the fifth multi-head attention module; The output of the third multi-head attention module is connected to the input of the first summing module; the output of the fourth multi-head attention module is connected to the input of the second summing module; the output of the fifth multi-head attention module is connected to the input of the third summing module; The output end of the first summation module is connected to the input end of the first self-attention module; the output end of the first self-attention module is connected to the input end of the first feedforward network and the input end of the fourth summation module; the output end of the first feedforward network module is connected to the input end of the fourth summation module; the output end of the second summation module is connected to the input end of the second self-attention module; the output end of the second self-attention module is connected to the input end of the fifth summation module and the input end of the second feedforward network; the output end of the third summation module is connected to the input end of the third self-attention module; the output end of the third self-attention module is connected to the input end of the third feedforward network and the input end of the sixth summation module; the output end of the fourth summation module serves as the first output end of the trilinear multimodal fusion encoder; The output end of the fifth summation module serves as the second output end of the trilinear multimodal fusion encoder; The output end of the sixth summation module serves as the third output end of the trilinear multimodal fusion encoder.

2. A video-language representation learning method using the trilinear coding system of claim 1, characterized in that: The following steps are involved: S1. Use teaching video and TV datasets as data samples and divide them into video segments; S2. Use the dense caption generation model to extract dense captions for the video clips; extract the original ASR transcripts; and use the dense captions for each clip as supplementary text to the original ASR transcripts. S3, taking the video clip, the original ASR transcription text and the dense subtitle as the input of the trilinear encoding system, performs representation learning to obtain the video-language representation; S4. Use video-language representation for video-text multimodal information interaction.

3. A video-language representation learning method according to claim 2, characterized in that: The specific implementation of step S3 is as follows: S3-1, according to the formula: ( ) ( ) Get ASR transcribed text encoding and dense subtitle encoding ;in, 、 Respectively represent l Video clips with original ASR transcripts and dense captions; Indicates encoding of the ASR transcript of the video clip; Indicates encoding of dense subtitles; S3-2, according to the formula: ( ) Get video frame encoding ; Among them, Transformer represents encoding the context information of the video frame, Indicates the A sequence of video frames of a video clip; S3-3, using a trilinear multimodal fusion encoder to obtain video segment-level encoding; S3-4. Input the video clip-level encoding into the temporal transformer, learn the contextual information at the video clip level through pre-training pre-tasks, perform language representation learning, and obtain video-language representation.

4. A video-language representation learning method according to claim 3, characterized in that: The specific implementation of step S3-3 is as follows: S3-3-1. According to the formula: Get the ASR transcription text-video frame information interaction result ,in, Indicates the The intermediate representation of the layer is also called the hidden layer representation; u The first one represents the trilinear multimodal fusion encoder u layer; u The ASR transcript hidden layer representation and the u The dense caption hidden layer representations of the layers are and ; Represents query parameters; express The learnable weights of express The learnable weights of express The learnable weights of Indicates that the data is processed by the multi-head attention module; S3-3-2, according to the formula: Get the information interaction results between dense subtitles and video frames ,in, express The learnable weights of express The learnable weights of express The learnable weights of S3-3-3, according to the formula: = Get ASR transcript encoding with multimodal information ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of S3-3-4, according to the formula: = Get dense subtitle encoding ;in, for and The weighted sum of express The learnable weights of express The learnable weights of express The learnable weights of S3-3-5, according to the formula: = Get video frame encoding ;in, 、 、 The weighted sum of ; express The learnable weights of express The learnable weights of express The learnable weights of S3-3-6, according to the formula: Get the transcribed text encoding after processing by the self-attention module , dense caption encoding after self-attention module processing , video frame sequence encoding after processing by the self-attention module ;in, Indicates that the data is processed by the self-attention module; S3-3-7. The outputs of the three self-attention modules are processed by a feedforward network to obtain a multimodal feature encoding that includes inter-modal interaction information and intra-modal autocorrelation information. S3-3-8, the three obtained polymorphic feature codes are recorded as the original ASR transcribed text sequence, the dense subtitle sequence and the video frame sequence; S3-3-9. Determine whether the current number of iterations reaches the set number of multimodal fusion coding unit layers of the modal fusion encoder; if so, proceed to step S3-3-10; otherwise, use the currently obtained original ASR transcribed text sequence, dense subtitle sequence, and video frame sequence as inputs of the trilinear coding system and return to step S3-3-1; S3-3-10. Concatenate the latest multimodal feature codes containing inter-modal interaction information and intra-modal autocorrelation information to obtain a video segment-level code.

Images