Video text cross-modal retrieval method and device

Abstract

The present invention provides a video-text cross-modal retrieval method and device, belonging to the field of data processing. The method comprises: obtaining raw data from a target data set; wherein the raw data includes video data and its corresponding text data; preprocessing the raw data to obtain a multimodal feature embedding vector; performing feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include global video features and local video features; embedding learning is performed on the global video features, local video features, and text features respectively to obtain a video embedding space, a video local relationship embedding space, and a text embedding space; and performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space. The present invention can achieve cross-modal retrieval more accurately and efficiently by combining vision, text, and local relationships for spatial embedding learning.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video text cross-modal retrieval method and device. Background Art

[0002] With the rapid development of internet technology, video has become one of the primary carriers of information dissemination. The explosive growth of video content has made conventional search engines unable to meet people's ever-changing search needs. Consequently, video-text cross-modal search has emerged. Currently, there are three main cross-modal search methods: those based on cross-modal feature vector matching, cross-modal interactive search methods based on attention mechanisms, and aligned fusion search methods based on visual and textual features.

[0003] However, among current cross-modal retrieval methods, those based on cross-modal feature vector matching have the following problems: Traditional video-text cross-modal retrieval methods often ignore or under-focus on the spatiotemporal characteristics of the visual modality that the image modality lacks, limiting the mining of video modal information and thus affecting extraction accuracy; The extraction process overly focuses on global features and ignores local details, resulting in low retrieval accuracy. Cross-modal interactive retrieval methods based on attention mechanisms have the following problems: They primarily employ a single-scale, low-resolution attention mechanism interaction model between visual and textual features. This lacks fine-grained feature interaction during feature extraction, resulting in low processing efficiency and insufficient precision; They may generate noise interference, negatively impacting retrieval performance and increasing computational costs. Aligned fusion retrieval methods based on visual and textual features have the following problems: insufficient utilization of fine-grained regional visual information, resulting in reduced matching accuracy; in cross-modal fusion of video and text, the entanglement mechanism between different modalities is more complex than the attention mechanism cross-modal interactive retrieval; the computational cost of using graph convolutional networks (GCNs) to construct a complete spatiotemporal relationship graph is too high and the efficiency is low.

[0004] Therefore, a cross-modal retrieval method is urgently needed to solve the current problems of low accuracy, low efficiency and high cost in cross-modal retrieval. Summary of the Invention

[0005] Embodiments of the present invention provide a video text cross-modal retrieval method and device to solve the problems of low accuracy, low efficiency and high cost in cross-modal retrieval at the current stage.

[0006] In a first aspect, an embodiment of the present invention provides a video text cross-modal retrieval method, comprising:

[0007] Obtaining original data in the target data set; wherein the original data includes video data and its corresponding text data;

[0008] Preprocess the original data to obtain the multimodal feature embedding vector;

[0009] Perform feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features;

[0010] The video global features, video local features and text features are embedded and learned respectively to obtain the video embedding space, video local relationship embedding space and text embedding space;

[0011] Video-text cross-modal retrieval is performed based on video embedding space, video local relationship embedding space and text embedding space.

[0012] In one possible implementation, embedding learning is performed on local features of the video to obtain a local relationship embedding space of the video, including:

[0013] The candidate region with the highest detection confidence in any frame of the video local features is used as the visual representation of the frame;

[0014] Based on visual representation, the data flow between entities contained in each frame of the video local features is processed;

[0015] Based on visual representation, the data flow between each frame of the video local features is processed;

[0016] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0017] In one possible implementation, based on visual representation, the data flow between entities contained in each frame of the local features of the video is processed, including:

[0018] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0019] Inputting the text representation into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data;

[0020] The visual representation used to represent the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain the first fused representation;

[0021] Passing the fused first representation through the feedforward network in the jump connection fusion module to obtain the first text representation with visual modality attention;

[0022] Inputting the first text representation with visual modal attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module;

[0023] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation between each frame in the video local feature and the overall video local feature.

[0024] In one possible implementation, based on visual representation, the data stream between each frame of the local features of the video is processed, including:

[0025] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0026] At the frame level, the text representation is input into the self-attention layer of any skip connection fusion module to obtain the text representation after the self-attention layer.

[0027] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation;

[0028] The fused second representation is passed through the feedforward network in the jump connection fusion module to obtain the second text representation with visual modality attention;

[0029] Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module;

[0030] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each second multimodal representation is residually connected to obtain video local relationship information including spatiotemporal information.

[0031] In one possible implementation, a video local relationship embedding space is obtained based on the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, including:

[0032] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are aggregated to obtain the local relationship embedding space of the video.

[0033] In one possible implementation, video-text cross-modal retrieval is performed based on the video embedding space, the video local relationship embedding space, and the text embedding space, including:

[0034] Mapping the video embedding space, the video local relationship embedding space, and the text embedding space into the joint embedding space for joint embedding learning, and obtaining the visual-text cross-modal features after joint embedding learning;

[0035] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0036] In one possible implementation, the video embedding space, the video local relationship embedding space, and the text embedding space are mapped into a joint embedding space for joint embedding learning. The visual-text cross-modal features obtained after the joint embedding learning include:

[0037] In the joint embedding space, the global video-text and local relationship-text are obtained based on the video embedding space, the video local relationship embedding space and the text embedding space. The cosine similarity between the global video-text and the local relationship-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

[0038] In one possible implementation, the raw data is preprocessed to obtain a multimodal feature embedding vector, including:

[0039] Extract key frames from video data;

[0040] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0041] In one possible implementation, feature extraction is performed on the multimodal feature embedding vector to obtain video features and text features, including:

[0042] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data;

[0043] Combine 2D appearance and 3D motion features to obtain the global features of the video;

[0044] Perform part-of-speech extraction on the multimodal feature embedding vector, generate a corresponding semantic relationship graph based on the part-of-speech extraction result, and obtain local features of the video based on the semantic relationship graph;

[0045] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0046] In a second aspect, an embodiment of the present invention provides a video text cross-modal retrieval device, comprising:

[0047] The acquisition module is used to obtain the original data in the target data set; wherein the original data includes video data and its corresponding text data;

[0048] The preprocessing module is used to preprocess the raw data to obtain a multimodal feature embedding vector;

[0049] An extraction module is used to extract features from the multimodal feature embedding vector to obtain video features and text features; wherein the video features include global video features and local video features;

[0050] The learning module is used to embed the video global features, video local features and text features respectively to obtain the video embedding space, video local relationship embedding space and text embedding space;

[0051] The retrieval module is used to perform cross-modal video and text retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space.

[0052] The embodiments of the present invention provide a method and device for cross-modal retrieval of video text. Compared with traditional methods, the embodiments of the present invention can significantly improve the efficiency and accuracy of cross-modal retrieval and reduce subsequent computing costs by preprocessing the original data. In addition, since the current mining of video spatiotemporal features is not in-depth enough or the effective integration with other modalities is ignored, the accuracy of cross-modal retrieval is limited. The embodiments of the present invention particularly emphasize the unique spatiotemporal characteristics of the video in addition to visual features and text features, that is, the local embedding space of the video, and effectively integrate it with the visual and textual modalities. Finally, by combining vision, text and local relationships for spatial embedding learning, cross-modal retrieval can be achieved more accurately and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of the implementation of the video text cross-modal retrieval method provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a hierarchical graphical interpretation model provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a cross-modal jump connection network provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of the spatiotemporal feature fusion model provided by an embodiment of the present invention;

[0058] Figure 5 A diagram illustrating the structure of a video-text cross-modal retrieval method according to an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the structure of a video-text cross-modal retrieval device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0062] Figure 1 The implementation flow chart of the video text cross-modal retrieval method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0063] Step 110: Acquire original data in the target data set; wherein the original data includes video data and its corresponding text data.

[0064] Step 120: Preprocess the original data to obtain a multimodal feature embedding vector.

[0065] In this embodiment, in the traditional alignment and fusion retrieval method based on visual and text features, there is a problem of insufficient utilization of fine-grained regional visual information, which leads to reduced matching accuracy. Therefore, before performing cross-modal retrieval of video text, this embodiment preprocesses the original data to perform fine-grained video-text alignment on the original data before feature extraction, thereby improving feature extraction efficiency.

[0066] Accordingly, in an optional embodiment, preprocessing the original data to obtain a multimodal feature embedding vector may include:

[0067] Extract key frames from video data.

[0068] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0069] In this embodiment, the video data and text data in the original data can be input into the model to achieve pre-training of the original data before feature extraction and fusion, extracting interactive visual / text features to obtain high-quality video-text data and improve the efficiency and accuracy of subsequent feature extraction. Accordingly, the pre-training model can be the Align and Prompt video-language pre-training model (ALPRO).

[0070] To meet the input requirements of the pre-trained model, keyframe extraction is required. Keyframes are representative single images in a video stream that capture important scenes or temporal changes within the video. They are important static images used for content summarization and retrieval. To ensure a relatively uniform distribution of frames across all moments in the video, embodiments of the present invention utilize a fixed sampling rate for keyframe extraction. This fixed sampling rate can be customized as needed.

[0071] After extracting the keyframes, the keyframes and their corresponding text data are fed into the pre-trained model. In the pre-trained model, a video encoder with a 12-layer TimeSformer is used to extract video features from the keyframes. The specific steps are as follows:

[0072] First, each keyframe image is segmented into multiple non-overlapping small regions. A linear projection layer is used to generate a sequence of word units from these small regions, and a learnable positional embedding is inserted. A self-attention mechanism is then applied to both the temporal and spatial dimensions. A temporal fusion layer (average pooling) is then added to incorporate the temporal features of each frame into the video features. Consequently, the output of the video encoder is a sequence of visual embeddings.

[0073] The text data corresponding to each keyframe image is passed through a text encoder with a six-layer Transformer to output a text embedding sequence. The encoder's video and text embedding sequences are then processed through a visual-text contrast loss function (VTC), which reduces the distance between positive pairings and increases the distance between negative pairings, thereby strengthening cross-modal matching between video and text.

[0074] Since position embeddings have already been injected into the video and text encoders, the video and text feature embedding vectors can be directly concatenated. A multimodal encoder consisting of three Transformer layers performs cross-modal interaction, outputting a pre-trained multimodal embedding space and multimodal feature embedding vectors for subsequent feature extraction. The video and text data passing through the encoder will pass through a multimodal encoding / decoding layer that includes a cross-attention mechanism, and will be learned through the visual-text matching (VTM) loss function and the language model (LM) loss function to obtain higher-quality datasets and improve matching accuracy.

[0075] After training with the ALPRO model, local and global alignment between video and text modalities can be easily achieved. Furthermore, pre-training the model on raw data yields more fine-grained details, facilitating video-text feature input and matching.

[0076] Step 130: Perform feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features.

[0077] In this embodiment, based on multi-level analysis, the multimodal feature embedding vector can be accurately decomposed into two dimensions: video features and text features, aiming to achieve extraction of global-local and coarse-fine granularity representations.

[0078] Accordingly, in an optional embodiment, performing feature extraction on the multimodal feature embedding vector to obtain video features and text features may include:

[0079] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data.

[0080] Combining 2D appearance and 3D motion features to obtain the global features of the video.

[0081] Parts of speech are extracted from the multimodal feature embedding vector, and the corresponding semantic relationship graph is generated based on the part-of-speech extraction results, and the local features of the video are obtained based on the semantic relationship graph.

[0082] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0083] In this embodiment, video features are extracted using a multi-level approach:

[0084] At the global level, the multimodal feature embedding vectors obtained from the pre-trained model are respectively passed through a 2D Convolutional Neural Network (2D-CNN) and a 3D Convolutional Neural Network (3D-CNN) to extract 2D appearance and 3D motion features from the video data as a multi-dimensional global representation of the video data. The 2D appearance and 3D motion features are combined to achieve in-depth mining of video frame-level features at the global level. The combined 2D appearance and 3D motion features are then aggregated through a maximum pooling layer to obtain the global video features.

[0085] At the fine-grained local level, part-of-speech extraction is performed on the multimodal feature embedding vector from a multi-layered perspective. Part-of-speech extraction can be performed by breaking down actions into verbs and nouns, while simultaneously generating a semantic relationship graph corresponding to the text description. This allows for in-depth mining of fine-grained local video features, and local video features are derived based on the semantic relationship graph.

[0086] The extraction of text features also follows a multi-level strategy, which is divided into two levels: hierarchical graph extraction and text part-of-speech extraction.

[0087] Figure 2 The structural diagram of the hierarchical graphic interpretation model provided by the embodiment of the present invention is shown below. Figure 2 The hierarchical graphic extraction in this embodiment will be described.

[0088] In this embodiment, hierarchical graph extraction mainly adopts a hierarchical graph interpretation model (Hierarchical Graph Resoning, HGR), which divides the text description into three levels: event, action, and object. Under this multi-level framework, the entire sentence represents a global event, which is composed of a series of actions under the temporal dimension, and each action is composed of different entities. Actions correspond to verbs in the text description, and entities correspond to nouns in the text description. The content encoding of these three levels of features forms a word vector embedding, and a semantic role relationship graph is constructed for the embedded vectors. The graph convolutional neural network (Graph Convolutional Networks, GCN) is used in combination with the attention mechanism to assign different weights to different semantic roles in the semantic role relationship graph, perform representation enhancement, correspond to the text part of speech, and facilitate subsequent processing.

[0089] In this embodiment, to extract text parts of speech, the multimodal feature embedding vector is first segmented into text description tokens, and each word is tagged with a part-of-speech tag, such as noun, verb, or adjective. Using a cross-modal embedding network, the corresponding word is placed into the embedding network for the corresponding part of speech. After encoding through the embedding network, a new text representation with part of speech is formed. This multi-level text feature extraction enhances the granularity and richness of the text representation, making the module's understanding of semantic information more accurate and comprehensive.

[0090] Finally, the text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0091] Step 140: Embedding learning is performed on the video global features, video local features and text features respectively to obtain a video embedding space, a video local relationship embedding space and a text embedding space.

[0092] In this embodiment, in the feature extraction part of the above-mentioned related embodiments, the basic features of the text have been effectively jointly mined in a fine-grained manner and the text embedding space has been output. Therefore, this part will not perform too much processing on the text embedding space. In terms of embedding learning of global video features, 2D-CNN can be used to extract the appearance features of key frames, and 3D-CNN can be used to extract action features. These features are then spliced ​​again accordingly, and a point-by-point linear layer is applied to obtain the global features of the video. Subsequently, the obtained global features of the video are processed through a multi-layer Transformer and an aggregation layer with an attention mechanism to form a video embedding space.

[0093] To improve cross-modal extraction performance and accuracy, embedding learning is performed not only on global video features but also on local video features. Since spatiotemporal characteristics are crucial for video, this allows for deeper exploration of the relationships between the entire video and its parts, as well as between parts. By establishing spatiotemporal relationships between entities and using similarity learning, the model can identify subtle differences brought about by differences in spatiotemporal video features at a finer level, improving retrieval accuracy.

[0094] Accordingly, in an optional embodiment, embedding learning is performed on local features of the video to obtain a local relationship embedding space of the video, which may include:

[0095] The candidate region with the highest detection confidence in any frame of the video local features is used as the visual representation of the frame.

[0096] Based on visual representation, the data flow between entities contained in each frame of the video local features is processed.

[0097] Based on visual representation, the data flow between each frame of the video local features is processed.

[0098] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0099] In this embodiment, in order to construct visual relationships in the video, that is, the association between the whole and the parts, and between the parts, this embodiment first uses the pre-trained Faster RCNN model to detect local region features at the video frame level, and selects the N candidate regions with the highest detection confidence in each frame as the visual representation of each frame.

[0100] After obtaining the visual representation of each frame, the local relationship embedding space of the video needs to be obtained. Accordingly, the spatial information contained in the video is integrated through a cross-modal skip connection network. Then, the temporal features contained in the video and the local information of the original video frames (the visual representation corresponding to each frame) are aggregated using the Temporal Residual Transformer (Temporal-SRT) and residual connections. The above operations aim to capture higher-level visual relationships, thereby fully exploring the spatiotemporal characteristics of the video and conducting deep learning of visual modal representations.

[0101] Since SRT mainly operates on two data streams, one is the data stream between entities within the frame, and the other is the data stream between frames, that is, the data stream between entities contained in each frame of the video local features and the data stream between each frame of the video local features, accordingly, it is necessary to process these two data streams separately based on visual representation to obtain the video local relationship embedding space.

[0102] Accordingly, in an optional embodiment, processing the data flow between entities contained in each frame of the local video feature based on the visual representation may include:

[0103] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0104] The text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data.

[0105] The visual representation of the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain the first fused representation.

[0106] The fused first representation is passed through the feedforward network in the jump connection fusion module to obtain the first text representation with visual modality attention.

[0107] The first text representation with visual modal attention and the visual representation of the data flow between entities contained in each frame in the local features of the video are input into the cross-modal connection attention module in the skip connection fusion module to obtain the first multimodal representation corresponding to the skip connection fusion module.

[0108] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation between each frame in the video local feature and the overall video local feature.

[0109] Figure 3 A schematic diagram of the structure of a cross-modal jump connection network provided by an embodiment of the present invention is shown below in conjunction with Figure 3 This embodiment will be described.

[0110] In this embodiment, the cross-modal skip connection network consists of N skip connection fusion blocks, each of which contains S asymmetric joint attention layers and 1 cross-modal connection attention block. The skip connection network utilizes the efficiency of splicing cross-modal fusion and the efficiency of the asymmetric joint attention mechanism to enhance cross-modal spatiotemporal fusion in a recursive manner. Among them, the asymmetric joint attention layer consists of a self-attention (SA) layer, a cross-attention (CA) layer and a feedforward network (FFN). The cross-modal connection attention block (CAB) consists of a self-attention (SA) layer and a feedforward network (FFN).

[0111] To process the data flow between entities contained in each frame of the video local features, each skip connection fusion module in the cross-modal skip connection network can perform the following steps:

[0112] The text representation encoded by the text encoder is passed through the self-attention layer of any skip connection fusion module to obtain the text representation after the self-attention layer. The visual representation of the data flow between entities contained in each frame in the local features of the video is then injected into the text representation after the self-attention layer through the cross-attention layer, and the two are fused through the cross-attention layer to obtain the first fused representation. Thereafter, the fused first representation is passed through a feedforward network (FFN) to obtain the first text representation with visual modal attention. Finally, the first text representation with visual modal attention and the original visual representation (the visual representation of the data flow between entities contained in each frame in the local features of the video) are passed through the cross-modal connection attention module to obtain the first multimodal representation corresponding to the skip connection fusion module to establish a connection between visual and textual cross-modal entities.

[0113] The above operation is performed once for each skip connection fusion module, for a total of N times. In addition, a regularization layer operation (LN) is applied to each layer in each skip connection fusion module to improve model stability and convergence speed. Afterwards, residual connections are used on the N first multimodal representations to aggregate each frame of the local video features with the spatial information to determine the correlation between each frame of the local video features and the overall local video features.

[0114] Next, the data stream between frames is processed. Accordingly, in some optional embodiments, the data stream between each frame of the local features of the video is processed based on the visual representation, which may include:

[0115] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0116] In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer.

[0117] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation.

[0118] The fused second representation is passed through the feedforward network in the jump connection fusion module to obtain the second text representation with visual modality attention.

[0119] The second text representation with visual modal attention and the visual representation used to represent each frame in the local features of the video are input into the cross-modal connection attention module in the skip connection fusion module to obtain the second multimodal representation corresponding to the skip connection fusion module.

[0120] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each second multimodal representation is residually connected to obtain video local relationship information including spatiotemporal information.

[0121] In this embodiment, for the data flow between each frame of the local feature of the video, operations need to be performed at the frame level space, which is similar to the process of processing the data flow between entities contained in each frame of the local feature of the video. The time connection between frames can be established through the time residual Transformer, and then the time information can be aggregated using the residual connection to obtain the video local relationship information containing spatiotemporal information. The specific operations can refer to the process of processing the data flow between entities contained in each frame of the local feature of the video, which will not be repeated here.

[0122] In an optional embodiment, obtaining a video local relationship embedding space based on the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features may include:

[0123] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are aggregated to obtain the local relationship embedding space of the video.

[0124] In this embodiment, based on visual representation, the correlation between each frame of the video local features and the overall video local features, as well as the video local relationship information including spatiotemporal information, can be obtained. Using the attention feature aggregation layer, the two are feature aggregated to obtain the final video local relationship embedding space.

[0125] Step 150: Perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space.

[0126] In some optional embodiments, performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space may include:

[0127] The video embedding space, video local relationship embedding space and text embedding space are mapped to the joint embedding space for joint embedding learning, and the visual-text cross-modal features after joint embedding learning are obtained.

[0128] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0129] Figure 4 This is a schematic diagram of the structure of the spatiotemporal feature fusion model provided by the embodiment of the present invention. Figure 4 This embodiment will be described.

[0130] In this embodiment, the spatiotemporal feature fusion model is used to implement embedding learning of video global features, video local features and text features respectively, to obtain video embedding space, video local relationship embedding space and text embedding space, and to perform video text cross-modal retrieval based on video embedding space, video local relationship embedding space and text embedding space. In the spatiotemporal feature fusion model, after obtaining the video embedding space, video local relationship embedding space and text embedding space, the three are mapped to the joint embedding space for joint embedding learning, and the visual-text cross-modal features after joint embedding learning are obtained to achieve cross-modal fusion.

[0131] Specifically, in the joint embedding space, this process can include obtaining global video-text and local relationship-text based on the video embedding space, video local relationship embedding space, and text embedding space, calculating the cosine similarity of global video-text and local relationship-text, and calculating the hinge-based ternary ranking loss during the fusion process. By balancing hyperparameters, the cosine similarity metric between different modalities is optimized to improve the model's ability to identify positive and negative sample pairs, ensuring that the similarity of mutual matches is higher than any mismatched similarity, and enhancing the cross-modal understanding ability of the entire system. In addition, during the fusion process, the uniqueness of each modality is retained and complemented with other modal features through fusion learning to improve overall retrieval performance and accuracy.

[0132] As can be seen, before extracting features from video and text data, the embodiments of the present invention pre-train the raw data in the target dataset to obtain more fine-grained video and text modality data, that is, to obtain multimodal feature embedding vectors. The pre-training process involves aligning the video and text modalities, which can improve the efficiency of cross-modal retrieval while reducing computational costs, and this point should be protected.

[0133] Secondly, the embodiment of the present invention adopts a multi-level fine-grained extraction strategy for video features and text features. In terms of video features, the focus is on comprehensive extraction of global features and local features; in terms of text features, a hierarchical structure is formed, semantic role relationships are analyzed, and basic parts of speech are extracted. In addition, the local feature extraction of the video corresponds to the text feature extraction method. This method not only reflects the interactivity between the video and text modalities, but also facilitates subsequent feature fusion. The proposed multi-level fine-grained feature extraction method can achieve accurate cross-modal retrieval in video-text retrieval.

[0134] Finally, the embodiment of the present invention particularly emphasizes the unique spatiotemporal characteristics of the video in addition to visual features and text features, and effectively integrates them with visual and text modalities. By using a cross-modal jump connection network to fuse spatiotemporal features, the cross-attention mechanism and cross-modal connection attention block in the asymmetric joint attention layer in the cross-modal jump connection network can be utilized, and the effectiveness of splicing cross-modal fusion and the efficiency of the asymmetric joint attention layer can be utilized to enhance the fusion of visual and textual cross-modal representations. By using Transformer technology to fuse temporal features, the powerful representation and interpretation capabilities of Transformer can be utilized to greatly reduce computational costs. With the help of the self-attention mechanism of Transformer, it is possible to process variable-length input representations and act as a fully connected layer, and the weights of the fully connected layer can be dynamically generated from the representation relationship. The application of this technology greatly improves fusion efficiency and module flexibility. By combining vision, text and local relationships for spatial embedding learning, cross-modal retrieval can be achieved more accurately and efficiently.

[0135] Figure 5 The scheme structure diagram of the video text cross-modal retrieval method provided by the embodiment of the present invention is as follows: Figure 5 shown.

[0136] Considering the following problems with retrieval methods based on cross-modal feature vector matching: 1) For video-text modal data encoding containing complex structures, after converting and mapping them into a single vector space, it is impossible to clearly represent the inherent structure and cross-modal correlation, and direct and effective comparison cannot be performed; 2) Traditional video-text cross-modal retrieval methods are mostly based on image-text retrieval technology, mainly focusing on basic feature matching between visual and text modalities. However, the visual modality has spatiotemporal characteristics that the image modality does not have, and traditional retrieval methods often ignore or pay insufficient attention to them, limiting the mining of video modal information, thereby affecting the extraction accuracy; 3) The extraction process over-focuses on global features and ignores local details, resulting in low retrieval accuracy.

[0137] Cross-modal interactive retrieval methods based on attention mechanisms have the following problems: 1) In video-text datasets, the scale of video-text pairs is small, or although the number is sufficient, the feature diversity is lacking, which limits the generalization ability of the model; 2) The attention mechanism interaction model between visual and text features is mainly adopted at a single scale and low resolution. When extracting features, there is a lack of fine-grained feature interaction, resulting in low processing efficiency and insufficient accuracy; 3) Noise interference may be generated, which has a negative impact on retrieval performance and increases computational costs.

[0138] The aligned fusion retrieval method based on visual and textual features has the following problems: 1) Insufficient utilization of fine-grained regional visual information leads to reduced matching accuracy; 2) In the cross-modal fusion of video and text, the entanglement mechanism between different modalities is more complicated than the attention mechanism cross-modal interactive retrieval; 3) The computational cost of using graph convolutional networks (GCNs) to construct a complete spatiotemporal relationship graph is too high and the efficiency is low.

[0139] Therefore, an embodiment of the present invention proposes a method to solve the following technical problems: (1) How to extract multi-level, fine-grained video and text features to obtain global and local modal representation information, perform cross-modal interactive learning on the multi-level representation information, and improve retrieval accuracy; (2) How to fully explore the unique spatiotemporal characteristics of videos and improve the model's ability to understand dynamic content; (3) How to comprehensively utilize the advantages of multiple existing video-text retrieval methods to improve retrieval efficiency and accuracy, thereby optimizing overall retrieval performance.

[0140] Please refer to Figure 5The method provided by the embodiment of the present invention can be mainly implemented through the following three modules: a pre-training module based on ALPRO, a video-text feature extraction module based on multi-level analysis, and a spatiotemporal feature fusion module.

[0141] The ALPRO-based pre-training module performs fine-grained video-text alignment on the raw data in the target dataset (i.e., video data and its corresponding text data) before feature extraction to improve feature extraction efficiency. Furthermore, the ALPRO-based pre-training module extracts keyframes from the video data; these keyframes and their corresponding text data are input into the pre-trained model to generate multimodal feature embedding vectors.

[0142] The multi-level analysis-based video-text feature extraction module is used to accurately decompose the multimodal embedding vector obtained after processing the pre-trained model into two dimensions: visual and textual features, based on multi-level analysis. This aims to achieve the extraction of global-local and coarse-fine granularity representations. Accordingly, the multi-level analysis-based video-text feature extraction module is used to extract features from the multimodal feature embedding vector, obtaining video features and text features; video features include global video features and local video features.

[0143] The spatiotemporal feature fusion module is used to effectively and finely mine video and text features through a customized spatiotemporal feature fusion module and joint embedding learning technology, thereby improving the model retrieval accuracy and efficiency while reducing the model complexity and computational cost.

[0144] For parts that are not fully described in this embodiment, reference may be made to other related embodiments.

[0145] In summary, the embodiments of the present invention have the following advantages:

[0146] 1) Most existing work directly processes the dataset as input, or only performs basic preprocessing. However, datasets processed in this way often suffer from low quality and lack of feature diversity, which affects the generalization ability of the model and may limit the extraction efficiency and accuracy. In the embodiment of the present invention, before feature extraction, the video data and text data in the original data are first deeply pre-trained. Through modal alignment, more fine-grained modal data is obtained before retrieval, thereby significantly improving the efficiency and accuracy of cross-modal retrieval and reducing subsequent computing costs.

[0147] 2) Existing work does not fully explore fine-grained visual and textual features, and is usually limited to fine-grained feature extraction of a single modality, either video or text. In addition, current methods mostly focus on hierarchical extraction of a single modality and fail to achieve hierarchical interaction across modalities. The embodiments of the present invention combine fine-grained feature extraction with a hierarchical method based on semantic graphs, adopting hierarchical and fine-grained extraction methods for both video and text modalities, and realizing interaction between modalities during the extraction process, effectively improving training efficiency and retrieval accuracy.

[0148] 3) Existing work does not delve deeply enough into the spatiotemporal features of videos, or ignores effective fusion with other modalities, thus limiting the accuracy of cross-modal retrieval. In addition, existing spatiotemporal fusion methods are inefficient and computationally expensive. The embodiments of the present invention make full use of the unique spatiotemporal characteristics of videos and adopt an innovative spatiotemporal feature fusion method based on a cross-modal jump connection network and a residual Transformer. While reducing computational costs, it achieves efficient and deep fusion of visual and textual modalities, comprehensively captures and utilizes information about video content, and improves the accuracy of cross-modal retrieval.

[0149] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0150] The following are device embodiments of the present invention. For details not fully described therein, reference may be made to the corresponding method embodiments described above.

[0151] Figure 6 A schematic diagram of the structure of a video text cross-modal retrieval device provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are detailed as follows:

[0152] like Figure 6 As shown, the video text cross-modal retrieval device 6 includes:

[0153] The acquisition module 61 is used to obtain the original data in the target data set; wherein the original data includes video data and its corresponding text data;

[0154] A preprocessing module 62 is used to preprocess the original data to obtain a multimodal feature embedding vector;

[0155] An extraction module 63 is used to extract features from the multimodal feature embedding vector to obtain video features and text features; wherein the video features include global video features and local video features;

[0156] A learning module 64 is configured to perform embedding learning on the video global features, the video local features, and the text features, respectively, to obtain a video embedding space, a video local relationship embedding space, and a text embedding space;

[0157] The retrieval module 65 is used to perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space and the text embedding space.

[0158] In one possible implementation, the learning module 64 is specifically configured to:

[0159] The candidate region with the highest detection confidence in any frame of the video local features is used as the visual representation of the frame;

[0160] Based on visual representation, the data flow between entities contained in each frame of the video local features is processed;

[0161] Based on visual representation, the data flow between each frame of the video local features is processed;

[0162] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0163] In one possible implementation, the learning module 64 is specifically configured to:

[0164] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0165] Inputting the text representation into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data;

[0166] The visual representation used to represent the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain the first fused representation;

[0167] Passing the fused first representation through the feedforward network in the jump connection fusion module to obtain the first text representation with visual modality attention;

[0168] Inputting the first text representation with visual modal attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module;

[0169] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation between each frame in the video local feature and the overall video local feature.

[0170] In one possible implementation, the learning module 64 is specifically configured to:

[0171] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0172] At the frame level, the text representation is input into the self-attention layer of any skip connection fusion module to obtain the text representation after the self-attention layer.

[0173] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation;

[0174] The fused second representation is passed through the feedforward network in the jump connection fusion module to obtain the second text representation with visual modality attention;

[0175] Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module;

[0176] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each second multimodal representation is residually connected to obtain video local relationship information including spatiotemporal information.

[0177] In one possible implementation, the learning module 64 is specifically configured to:

[0178] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are aggregated to obtain the local relationship embedding space of the video.

[0179] In a possible implementation, the retrieval module 65 is specifically configured to:

[0180] Mapping the video embedding space, the video local relationship embedding space, and the text embedding space into the joint embedding space for joint embedding learning, and obtaining the visual-text cross-modal features after joint embedding learning;

[0181] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0182] In a possible implementation, the retrieval module 65 is specifically configured to:

[0183] In the joint embedding space, the global video-text and local relationship-text are obtained based on the video embedding space, the video local relationship embedding space and the text embedding space. The cosine similarity between the global video-text and the local relationship-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

[0184] In a possible implementation, the preprocessing module 62 is specifically configured to:

[0185] Extract key frames from video data;

[0186] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0187] In a possible implementation, the extraction module 63 is specifically configured to:

[0188] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data;

[0189] Combine 2D appearance and 3D motion features to obtain the global features of the video;

[0190] Perform part-of-speech extraction on the multimodal feature embedding vector, generate a corresponding semantic relationship graph based on the part-of-speech extraction result, and obtain local features of the video based on the semantic relationship graph;

[0191] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0192] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0193] Those skilled in the art will appreciate that the templates, units, and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0194] If the module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various video text cross-modal retrieval method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc.

[0195] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A video text cross-modal retrieval method, characterized by: include: Obtaining original data in a target data set; wherein the original data includes video data and its corresponding text data; Preprocessing the raw data to obtain a multimodal feature embedding vector; Performing feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features; Embedding learning is performed on the video global features, the video local features, and the text features respectively to obtain a video embedding space, a video local relationship embedding space, and a text embedding space; Performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space; The embedding learning of the local features of the video to obtain the local relationship embedding space of the video includes: Taking the candidate region with the highest detection confidence in any frame of the local features of the video as the visual representation of the frame; Based on the visual representation, processing the data flow between entities contained in each frame of the local feature of the video; Based on the visual representation, processing the data stream between each frame of the local feature of the video; Obtaining a video local relationship embedding space according to the processed data flows between entities contained in each frame of the video local feature and the data flows between each frame of the video local feature; The processing of the data flow between entities contained in each frame of the local feature of the video based on the visual representation includes: For each skip connection fusion module in the cross-modal skip connection network, perform the following steps: Inputting the text representation into a self-attention layer in any skip connection fusion module to obtain a text representation after passing through the self-attention layer; wherein the text representation is determined based on the text data; Fusing the visual representation of the data flow between entities contained in each frame in the local video features with the text features after passing through the self-attention layer to obtain a fused first representation; Passing the fused first representation through the feedforward network in the skip connection fusion module to obtain a first text representation with visual modality attention; Inputting the first text representation with visual modality attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module; After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation between each frame in the local feature of the video and the overall local feature of the video.

2. The video text cross-modal retrieval method according to claim 1, characterized in that: The processing of the data stream between each frame of the local feature of the video based on the visual representation includes: For each skip connection fusion module in the cross-modal skip connection network, perform the following steps: In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after passing through the self-attention layer; Fusing the visual representation used to represent each frame in the local features of the video with the text features after passing through the self-attention layer to obtain a fused second representation; Passing the fused second representation through the feedforward network in the skip connection fusion module to obtain a second text representation with visual modality attention; Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module; After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each second multimodal representation is residually connected to obtain video local relationship information including spatiotemporal information.

3. The video text cross-modal retrieval method according to claim 2, characterized in that: The method of obtaining a video local relationship embedding space based on the processed data flows between entities contained in each frame of the video local feature and the data flows between each frame of the video local feature comprises: The correlation relationship between each frame in the video local feature and the overall video local feature and the video local relationship information including spatiotemporal information are feature aggregated to obtain a video local relationship embedding space.

4. The video text cross-modal retrieval method according to claim 1, characterized in that: The performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space includes: Mapping the video embedding space, the video local relationship embedding space, and the text embedding space into a joint embedding space for joint embedding learning, thereby obtaining visual-text cross-modal features after joint embedding learning; The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output result of the cross-modal encoder is used for video-text cross-modal retrieval.

5. The video text cross-modal retrieval method according to claim 4, characterized in that: Mapping the video embedding space, the video local relationship embedding space, and the text embedding space into a joint embedding space for joint embedding learning, and obtaining visual-text cross-modal features after joint embedding learning, includes: In the joint embedding space, global video-text and local relationship-text are obtained based on the video embedding space, the video local relationship embedding space and the text embedding space, the cosine similarity between the global video-text and the local relationship-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

6. The video text cross-modal retrieval method according to claim 1, characterized in that: The preprocessing of the original data to obtain a multimodal feature embedding vector includes: Extracting key frames of the video data; The key frames and their corresponding text data are input into a pre-trained model to obtain a multimodal feature embedding vector.

7. The video text cross-modal retrieval method according to claim 1, characterized in that: The extracting features of the multimodal feature embedding vector to obtain video features and text features includes: Passing the multimodal feature embedding vector through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain 2D appearance and 3D action features in the video data; Combining the 2D appearance and the 3D motion features to obtain a global video feature; Performing part-of-speech extraction on the multimodal feature embedding vector, generating a corresponding semantic relationship graph based on the part-of-speech extraction result, and obtaining local features of the video based on the semantic relationship graph; Hierarchical graph extraction and text part-of-speech extraction are adopted for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

8. A video text cross-modal retrieval device, characterized in that: include: An acquisition module is used to obtain the original data in the target data set; wherein the original data includes video data and its corresponding text data; A preprocessing module, configured to preprocess the raw data to obtain a multimodal feature embedding vector; An extraction module, configured to perform feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include global video features and local video features; A learning module is used to respectively perform embedding learning on the video global features, the video local features, and the text features to obtain a video embedding space, a video local relationship embedding space, and a text embedding space; A retrieval module, configured to perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space; The learning module is specifically used to: Taking the candidate region with the highest detection confidence in any frame of the local features of the video as the visual representation of the frame; Based on the visual representation, processing the data flow between entities contained in each frame of the local feature of the video; Based on the visual representation, processing the data stream between each frame of the local feature of the video; Obtaining a video local relationship embedding space according to the processed data flows between entities contained in each frame of the video local feature and the data flows between each frame of the video local feature; The learning module is specifically used to: For each skip connection fusion module in the cross-modal skip connection network, perform the following steps: Inputting the text representation into a self-attention layer in any skip connection fusion module to obtain a text representation after passing through the self-attention layer; wherein the text representation is determined based on the text data; Fusing the visual representation of the data flow between entities contained in each frame in the local video features with the text features after passing through the self-attention layer to obtain a fused first representation; Passing the fused first representation through the feedforward network in the skip connection fusion module to obtain a first text representation with visual modality attention; Inputting the first text representation with visual modality attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module; After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation between each frame in the local feature of the video and the overall local feature of the video.

Images