Video text cross-modal retrieval method and device

Abstract

The invention provides a video text cross-modal retrieval method and device, and belongs to the field of data processing. The method comprises the following steps: acquiring original data in a target data set; wherein the original data comprises video data and text data corresponding to the video data; preprocessing the original data to obtain a multi-modal feature embedding vector; performing feature extraction on the multi-modal feature embedding vector to obtain video features and text features; wherein the video features comprise video global features and video local features; performing embedding learning on the video global features, the video local features and the text features to obtain a video embedding space, a video local relation embedding space and a text embedding space; and performing video text cross-modal retrieval based on the video embedding space, the video local relation embedding space and the text embedding space. Spatial embedding learning is carried out by combining vision, texts and local relations, and cross-modal retrieval can be realized more accurately and efficiently.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video text cross-modal retrieval method and device. Background Art

[0002] With the rapid development of Internet technology, video has become one of the main carriers of information dissemination. The explosive growth of video content has made conventional retrieval unable to meet people's ever-changing search needs. Therefore, video-text cross-modal retrieval has emerged. At present, there are three main methods of cross-modal retrieval: a retrieval method based on cross-modal feature vector matching, a cross-modal interactive retrieval method based on attention mechanism, and an aligned fusion retrieval method based on visual and text features.

[0003] However, among the current cross-modal retrieval methods, the retrieval methods based on cross-modal feature vector matching have the following problems: the traditional video-text cross-modal retrieval methods often ignore or pay insufficient attention to the spatiotemporal characteristics of the visual modality that the image modality does not have, which limits the mining of video modality information and thus affects the extraction accuracy; the extraction process over-focuses on global features and ignores local details, resulting in low retrieval accuracy. The cross-modal interactive retrieval methods based on the attention mechanism have the following problems: they mainly adopt a single-scale, low-resolution attention mechanism interaction model between visual and text features. When extracting features, there is a lack of fine-grained feature interaction, resulting in low processing efficiency and insufficient accuracy; noise interference may be generated, which has a negative impact on retrieval performance and increases computational costs. The aligned fusion retrieval method based on visual and text features has the following problems: insufficient utilization of fine-grained regional visual information, resulting in reduced matching accuracy; in the fused video-text cross-modality, the entanglement mechanism between different modalities is more complicated than the attention mechanism cross-modal interactive retrieval; the computational cost of using graph convolutional network (GCN) when constructing a complete spatiotemporal relationship graph is too high and the efficiency is low.

[0004] Therefore, there is an urgent need for a cross-modal retrieval method to solve the current problems of low accuracy, low efficiency and high cost in cross-modal retrieval. Summary of the invention

[0005] The embodiments of the present invention provide a video text cross-modal retrieval method and device to solve the problems of low accuracy, low efficiency and high cost in cross-modal retrieval at the current stage.

[0006] In a first aspect, an embodiment of the present invention provides a video text cross-modal retrieval method, comprising:

[0007] Obtaining original data in the target data set; wherein the original data includes video data and its corresponding text data;

[0008] Preprocess the original data to obtain a multimodal feature embedding vector;

[0009] Perform feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features;

[0010] The global features of the video, the local features of the video and the text features are embedded and learned respectively to obtain the video embedding space, the video local relationship embedding space and the text embedding space;

[0011] Video-text cross-modal retrieval is performed based on video embedding space, video local relationship embedding space and text embedding space.

[0012] In a possible implementation, embedding learning is performed on the local features of the video to obtain the local relationship embedding space of the video, including:

[0013] The candidate region with the highest detection confidence in any frame of the local features of the video is used as the visual representation of the frame;

[0014] Based on visual representation, the data flow between entities contained in each frame of the local features of the video is processed;

[0015] Based on visual representation, the data flow between each frame of the local features of the video is processed;

[0016] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0017] In a possible implementation, based on the visual representation, the data flow between entities contained in each frame of the local feature of the video is processed, including:

[0018] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0019] Inputting the text representation into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data;

[0020] The visual representation used to represent the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused first representation;

[0021] Passing the fused first representation through the feedforward network in the jump connection fusion module to obtain a first text representation with visual modality attention;

[0022] Inputting the first text representation with visual modality attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module;

[0023] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the first multimodal representations are residually connected to determine the correlation between each frame in the local feature of the video and the local feature of the video as a whole.

[0024] In a possible implementation, based on the visual representation, the data stream between each frame of the local feature of the video is processed, including:

[0025] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0026] In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer;

[0027] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation;

[0028] Passing the fused second representation through the feedforward network in the jump connection fusion module to obtain a second text representation with visual modality attention;

[0029] Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module;

[0030] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the second multimodal representations are residually connected to obtain video local relationship information including spatiotemporal information.

[0031] In a possible implementation, according to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, a video local relationship embedding space is obtained, including:

[0032] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are feature aggregated to obtain the local relationship embedding space of the video.

[0033] In a possible implementation, video-text cross-modal retrieval is performed based on a video embedding space, a video local relationship embedding space, and a text embedding space, including:

[0034] Mapping the video embedding space, the video local relationship embedding space and the text embedding space into the joint embedding space for joint embedding learning, and obtaining the visual-text cross-modal features after joint embedding learning;

[0035] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0036] In a possible implementation, the video embedding space, the video local relationship embedding space, and the text embedding space are mapped to the joint embedding space for joint embedding learning, and the visual-text cross-modal features after the joint embedding learning are obtained, including:

[0037] In the joint embedding space, the global video-text and local relation-text are obtained based on the video embedding space, the video local relation embedding space and the text embedding space. The cosine similarity between the global video-text and the local relation-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

[0038] In a possible implementation, the original data is preprocessed to obtain a multimodal feature embedding vector, including:

[0039] Extract key frames of video data;

[0040] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0041] In a possible implementation, feature extraction is performed on the multimodal feature embedding vector to obtain video features and text features, including:

[0042] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data;

[0043] Combine 2D appearance and 3D motion features to obtain the global features of the video;

[0044] Extract the part of speech from the multimodal feature embedding vector, generate the corresponding semantic relationship graph based on the part of speech extraction result, and obtain the local features of the video based on the semantic relationship graph;

[0045] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0046] In a second aspect, an embodiment of the present invention provides a video text cross-modal retrieval device, comprising:

[0047] The acquisition module is used to obtain the original data in the target data set; wherein the original data includes video data and its corresponding text data;

[0048] The preprocessing module is used to preprocess the original data to obtain a multimodal feature embedding vector;

[0049] An extraction module is used to extract features from the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features;

[0050] A learning module is used to embed and learn the video global features, the video local features and the text features respectively, and obtain the video embedding space, the video local relationship embedding space and the text embedding space;

[0051] The retrieval module is used for cross-modal retrieval of video and text based on video embedding space, video local relationship embedding space and text embedding space.

[0052] The embodiments of the present invention provide a video-text cross-modal retrieval method and device. Compared with the traditional method, the embodiments of the present invention can significantly improve the efficiency and accuracy of cross-modal retrieval and reduce the subsequent computing cost by preprocessing the original data. In addition, since the current mining of video spatiotemporal features is not deep enough or the effective integration with other modalities is ignored, the accuracy of cross-modal retrieval is limited. The embodiments of the present invention particularly emphasize the unique spatiotemporal characteristics of the video in addition to the visual features and text features, that is, the local embedding space of the video, and effectively integrate it with the visual and textual modalities. Finally, by combining vision, text and local relationships for spatial embedding learning, cross-modal retrieval can be achieved more accurately and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of the implementation of the video text cross-modal retrieval method provided by the embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a hierarchical graphic interpretation model provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a cross-modal jump connection network provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of a spatiotemporal feature fusion model provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of a video text cross-modal retrieval method provided by an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the structure of a video text cross-modal retrieval device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0062] Figure 1 The implementation flow chart of the video text cross-modal retrieval method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0063] Step 110: Acquire original data in the target data set; wherein the original data includes video data and its corresponding text data.

[0064] Step 120: Preprocess the original data to obtain a multimodal feature embedding vector.

[0065] In this embodiment, in the traditional alignment and fusion retrieval method based on visual and text features, there is a problem of insufficient utilization of fine-grained regional visual information, which leads to reduced matching accuracy. Therefore, before performing cross-modal retrieval of video text, this embodiment preprocesses the original data to perform fine-grained video-text alignment on the original data before feature extraction, thereby improving feature extraction efficiency.

[0066] Accordingly, in an optional embodiment, preprocessing the original data to obtain a multimodal feature embedding vector may include:

[0067] Extract key frames of video data.

[0068] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0069] In this embodiment, the video data and text data in the original data can be input into the model to realize pre-training of the original data before feature extraction and fusion, extract interactive visual / text features, obtain high-quality video-text data, and improve the efficiency and accuracy of subsequent feature extraction. Correspondingly, the pre-trained model can be an alignment and prompt video-language pre-trained model (Align and Prompt, ALPRO)

[0070] In order to meet the input requirements of the pre-trained model, it is necessary to first extract key frames from the video data; a key frame refers to a single image that is representative in the video stream and can capture important scenes or time changes in the video, and is an important static image in the video for content summary and retrieval. To ensure that the frames extracted at each moment of the video are relatively average, the embodiment of the present invention can use a fixed sampling rate to extract key frames, wherein the fixed sampling rate can be set as required.

[0071] After extracting the keyframes, the keyframes and their corresponding text data are input into the pre-trained model. In the pre-trained model, the video features in the keyframes are extracted through a video encoder with 12 layers of TimeSformer. The specific steps are as follows:

[0072] First, each image in the key frame is divided into multiple non-overlapping small areas, and a sequence of word units in the small image areas is generated through a linear projection layer, and a learnable position embedding is inserted. After that, the self-attention mechanism is applied in the time and space dimensions respectively, and then a time fusion layer (average pooling layer) is added to add the time dimension features of each frame to the video features. Correspondingly, the output of the video encoder is a visual embedding sequence.

[0073] The text data corresponding to each image in the key frame will be output through a text encoder with 6 layers of Transformer to output a text embedding sequence. The video embedding sequence and text embedding sequence after the encoder will be shortened and the negative pairing distance will be increased through the visual-text contrast loss function VTC, thus strengthening the cross-modal matching between video and text.

[0074] Since the position embedding has been injected into the video and text encoders, the video and text feature embedding vectors can be directly spliced, and cross-modal interaction can be performed through a multimodal encoder composed of 3 layers of Transformers, outputting the pre-trained multimodal embedding space and multimodal feature embedding vectors for subsequent feature extraction. The video data and text data passed through the encoder will be judged and learned through the multimodal encoding / decoding layer covering the cross-attention mechanism, through the visual-text matching (VTM) loss function and the language model (LM) loss function, in order to obtain higher quality data sets and improve matching accuracy.

[0075] After ALPRO model training, it is easier to achieve local and global alignment between video and text modalities. In addition, the original data can obtain more fine-grained details after pre-training with this model, which is helpful for video-text feature input and matching.

[0076] Step 130: extracting features from the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features.

[0077] In this embodiment, based on multi-level analysis, the multimodal feature embedding vector can be accurately decomposed into two dimensions: video features and text features, aiming to achieve extraction of global-local and coarse-fine granularity representations.

[0078] Accordingly, in an optional embodiment, feature extraction is performed on the multimodal feature embedding vector to obtain video features and text features, which may include:

[0079] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data.

[0080] Combining 2D appearance and 3D motion features, we can obtain the global features of the video.

[0081] Parts of speech are extracted from the multimodal feature embedding vector, and the corresponding semantic relationship graph is generated based on the part-of-speech extraction result, and the local features of the video are obtained based on the semantic relationship graph.

[0082] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0083] In this embodiment, the video features are extracted using a multi-level approach:

[0084] At the global level, the multimodal feature embedding vectors obtained by the pre-trained model are respectively passed through a 2D Convolutional Neural Network (2D-CNN) and a 3D Convolutional Neural Network (3D-CNN) to extract 2D appearance and 3D motion features from the video data as a multi-dimensional global representation of the video data. The 2D appearance and 3D motion features are combined to achieve in-depth mining of video frame-level features at the global level, and then the combined 2D appearance and 3D motion features are aggregated through the maximum pooling layer to obtain the global features of the video.

[0085] At the fine-grained local level, the multimodal feature embedding vector is deeply extracted from the part of speech from a multi-level perspective. Part of speech extraction can be: decomposing the action into verbs and nouns, and generating a semantic relationship graph corresponding to the text description to achieve deep mining of fine-grained local video features, and obtain local video features based on the semantic relationship graph.

[0086] The extraction of text features also follows a multi-level strategy, which is divided into two levels: hierarchical graph extraction and text part-of-speech extraction.

[0087] Figure 2 The structural diagram of the hierarchical graphic interpretation model provided by the embodiment of the present invention is shown below in conjunction with Figure 2 The hierarchical pattern extraction in this embodiment is described.

[0088] In this embodiment, hierarchical graph extraction mainly adopts a hierarchical graph interpretation model (Hierarchical Graph Resoning, HGR), which divides the text description into three levels: event, action, and entity. Under this multi-level framework, the entire sentence represents a global event, which is composed of a series of actions under the temporal dimension, and each action is composed of different entities. The action corresponds to the verb in the text description, and the entity corresponds to the noun in the text description. The content encoding of these three levels of features forms a word vector embedding, and a semantic role relationship diagram is constructed for the embedded vector. The graph convolutional neural network (Graph Convolutional Networks, GCN) is used in combination with the attention mechanism to assign different weights to different semantic roles in the semantic role relationship diagram, and the representation is enhanced, corresponding to the text part of speech, which is convenient for subsequent processing.

[0089] In this embodiment, in order to extract text part-of-speech, the multimodal feature embedding vector is first segmented into text description tokens, and each word is tagged with part-of-speech, such as nouns, verbs, adjectives, etc. Through the cross-modal embedding network, the corresponding words are placed in the embedding network of the corresponding part-of-speech, and a new text representation with part-of-speech is formed after embedding network encoding. Through multi-level text feature extraction, the fine-grainedness and richness of text representation are enhanced, making the module's understanding of semantic information more accurate and comprehensive.

[0090] Finally, the text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0091] Step 140: embedding learning is performed on the video global features, the video local features and the text features respectively to obtain a video embedding space, a video local relationship embedding space and a text embedding space.

[0092] In this embodiment, in the feature extraction part of the above-mentioned related embodiments, the basic features of the text have been effectively jointly mined in a fine-grained manner and the text embedding space has been output, so this part will no longer process the text embedding space too much. In terms of embedding learning of global video features, 2D-CNN can be used to extract the appearance features of key frames, and 3D-CNN can be used to extract action features. These features are then concatenated again accordingly, and a point-by-point linear layer is applied to obtain the global features of the video. Subsequently, the acquired global features of the video are processed through a multi-layer Transformer and an aggregation layer with an attention mechanism to form a video embedding space.

[0093] In order to improve the performance and accuracy of cross-modal extraction, in addition to embedding learning of global video features, local video features are also embedded. Since spatiotemporal characteristics are important characteristics of videos, they can more deeply explore the relationship between the whole and the local, as well as the relationship between the local and the local in the video. By establishing the spatiotemporal characteristic relationship between entities and similarity learning, the model can recognize the subtle differences caused by different spatiotemporal characteristics of videos at a finer level, thereby improving retrieval accuracy.

[0094] Accordingly, in an optional embodiment, embedding learning is performed on the local features of the video to obtain the local relationship embedding space of the video, which may include:

[0095] The candidate region with the highest detection confidence in any frame of the video local features is used as the visual representation of the frame.

[0096] Based on the visual representation, the data flow between entities contained in each frame of the local features of the video is processed.

[0097] Based on the visual representation, the data flow between each frame of the local features of the video is processed.

[0098] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0099] In this embodiment, in order to construct the visual relationship in the video, that is, the correlation between the whole and the part, and the local and the local in the video, this embodiment first uses the pre-trained Faster RCNN model to detect the local area features at the video frame level, and selects the N candidate areas with the highest detection confidence in each frame as the visual representation of each frame.

[0100] After obtaining the visual representation of each frame, it is necessary to obtain the local relationship embedding space of the video. Accordingly, the spatial information contained in the video is fused through the cross-modal jump connection network, and then the temporal features contained in the video and the local information of the original video frame (the visual representation corresponding to each frame) are aggregated using the temporal residual transformer (Temporal-SRT) and residual connection (Residual Connection). The above operations are designed to capture higher-level visual relationships, so as to fully explore the spatiotemporal characteristics of the video and conduct deep learning of the visual modality representation.

[0101] Since SRT mainly operates on two data streams, one is the data stream between entities within the frame, and the other is the data stream between frames, that is, the data stream between entities contained in each frame of the video local features and the data stream between each frame of the video local features, accordingly, these two data streams need to be processed separately based on visual representation to obtain the video local relationship embedding space.

[0102] Accordingly, in an optional embodiment, based on the visual representation, processing the data stream between entities contained in each frame of the local feature of the video may include:

[0103] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0104] The text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data.

[0105] The visual representation of the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused first representation.

[0106] The fused first representation is passed through the feedforward network in the jump connection fusion module to obtain the first text representation with visual modality attention.

[0107] The first text representation with visual modal attention and the visual representation of the data flow between entities contained in each frame in the local features of the video are input into the cross-modal connection attention module in the jump connection fusion module to obtain the first multimodal representation corresponding to the jump connection fusion module.

[0108] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the first multimodal representations are residually connected to determine the correlation between each frame in the local feature of the video and the local feature of the video as a whole.

[0109] Figure 3 A schematic diagram of the structure of a cross-modal jump connection network provided by an embodiment of the present invention is shown below in conjunction with Figure 3 This embodiment is described.

[0110] In this embodiment, the cross-modal jump connection network consists of N jump connection fusion blocks, each of which contains S asymmetric joint attention layers and 1 cross-modal connection attention block. The jump connection network utilizes the efficiency of splicing cross-modal fusion and the efficiency of the asymmetric joint attention mechanism to enhance cross-modal spatiotemporal fusion in a recursive manner. Among them, the asymmetric joint attention layer consists of a self-attention (SA) layer, a cross-attention (CA) layer and a feedforward network (FFN). The cross-modal connection attention block (CAB) consists of a self-attention (SA) layer and a feedforward network (FFN).

[0111] In order to process the data flow between entities contained in each frame of the local features of the video, each skip connection fusion module in the cross-modal skip connection network can perform the following steps:

[0112] The text representation encoded by the text encoder is passed through the self-attention layer of any skip connection fusion module to obtain the text representation after the self-attention layer. Then the visual representation of the data flow between entities contained in each frame in the local features of the video is injected into the text representation after the self-attention layer through the cross-attention layer, and the two are fused through the cross-attention layer to obtain the first fused representation. Thereafter, the fused first representation is passed through a feedforward network (FFN) to obtain the first text representation with visual modal attention. Finally, the first text representation with visual modal attention and the original visual representation (the visual representation of the data flow between entities contained in each frame in the local features of the video) are passed through the cross-modal connection attention module to obtain the first multimodal representation corresponding to the skip connection fusion module to establish a connection between visual and textual cross-modal entities.

[0113] The above operation is performed once for each jump connection fusion module, and is performed N times in total. In addition, in each jump connection fusion module, a regularization layer operation (LN) is used for each layer to improve the model stability and convergence speed. After that, residual connections are used for the N first multimodal representations to aggregate each frame in the local features of the video with the spatial information to determine the correlation between each frame in the local features of the video and the overall local features of the video.

[0114] Next, the data stream between frames is processed. Accordingly, in some optional embodiments, based on the visual representation, the data stream between each frame of the local feature of the video is processed, which may include:

[0115] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0116] In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer.

[0117] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation.

[0118] The fused second representation is passed through the feedforward network in the jump connection fusion module to obtain the second text representation with visual modality attention.

[0119] The second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video are input into the cross-modal connection attention module in the jump connection fusion module to obtain the second multimodal representation corresponding to the jump connection fusion module.

[0120] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the second multimodal representations are residually connected to obtain video local relationship information including spatiotemporal information.

[0121] In this embodiment, for the data flow between each frame of the local feature of the video, it is necessary to operate on the frame level space, which is similar to the process of processing the data flow between entities contained in each frame of the local feature of the video. The time connection between frames can be established through the time residual Transformer, and then the time information is aggregated using the residual connection to obtain the video local relationship information containing time and space information. The specific operation can refer to the process of processing the data flow between entities contained in each frame of the local feature of the video, which will not be repeated here.

[0122] In an optional embodiment, obtaining the video local relationship embedding space according to the data flow between entities contained in each frame of the processed video local feature and the data flow between each frame of the video local feature may include:

[0123] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are feature aggregated to obtain the local relationship embedding space of the video.

[0124] In this embodiment, based on visual representation, the correlation between each frame of the local features of the video and the local features of the video as a whole, as well as the local relationship information of the video including spatiotemporal information can be obtained. The attention feature aggregation layer is used to perform feature aggregation on the two to obtain the final video local relationship embedding space.

[0125] Step 150: Perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space.

[0126] In some optional embodiments, performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space, and the text embedding space may include:

[0127] The video embedding space, the video local relationship embedding space and the text embedding space are mapped to the joint embedding space for joint embedding learning, and the visual-text cross-modal features after joint embedding learning are obtained.

[0128] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0129] Figure 4 The schematic diagram of the structure of the spatiotemporal feature fusion model provided in the embodiment of the present invention is shown below. Figure 4 This embodiment is described.

[0130] In this embodiment, the spatiotemporal feature fusion model is used to implement embedding learning of video global features, video local features and text features respectively, to obtain video embedding space, video local relationship embedding space and text embedding space, and to perform cross-modal retrieval of video text based on video embedding space, video local relationship embedding space and text embedding space. In the spatiotemporal feature fusion model, after obtaining the video embedding space, video local relationship embedding space and text embedding space, the three are mapped to the joint embedding space for joint embedding learning, and the visual-text cross-modal features after joint embedding learning are obtained to achieve cross-modal fusion.

[0131] Specifically, in the joint embedding space, the process may include obtaining global video-text and local relationship-text based on the video embedding space, the video local relationship embedding space and the text embedding space, calculating the cosine similarity of the global video-text and the local relationship-text, and calculating the hinge-based ternary ranking loss during the fusion process, optimizing the cosine similarity metric between different modalities through the balance of hyperparameters, improving the model's ability to identify positive and negative sample pairs, ensuring that the similarity of mutual matching is higher than any unmatched similarity, and enhancing the cross-modal understanding ability of the entire system. In addition, during the fusion process, the uniqueness of each modality is retained, and it is complemented by other modal features through fusion learning to improve the overall retrieval performance and accuracy.

[0132] It can be seen that, first of all, before extracting features from video data and text data, the embodiment of the present invention first pre-trains the original data in the target data set to obtain more fine-grained video and text modality data, that is, to obtain a multimodal feature embedding vector. The pre-training process involves the alignment of video and text modalities, which can improve the efficiency of cross-modal retrieval and reduce computational costs, and this point should be protected.

[0133] Secondly, the embodiment of the present invention adopts a multi-level fine-grained extraction strategy for video features and text features. In terms of video features, the focus is on comprehensive extraction of global features and local features; in terms of text features, a hierarchical structure is formed, semantic role relationships are analyzed, and basic parts of speech are extracted. In addition, the local feature extraction of the video corresponds to the text feature extraction method. This method not only reflects the interactivity between the video-text modalities, but also facilitates subsequent feature fusion. The proposed multi-level fine-grained feature extraction method can achieve accurate cross-modal retrieval in video-text retrieval.

[0134] Finally, the embodiment of the present invention particularly emphasizes the unique spatiotemporal characteristics of the video in addition to the visual features and text features, and effectively integrates them with the visual and text modalities. Using a cross-modal jump connection network to fuse spatiotemporal features, the cross-attention mechanism in the asymmetric joint attention layer in the cross-modal jump connection network and the cross-modal connection attention block can be utilized to enhance the fusion of visual and text cross-modal representations by utilizing the effectiveness of splicing cross-modal fusion and the efficiency of the asymmetric joint attention layer. Using Transformer technology to fuse temporal features can greatly reduce the computational cost by utilizing the powerful representation and interpretation capabilities of Transformer. With the help of the self-attention mechanism of Transformer, it is possible to process input variable-length representations and act as a fully connected layer, and the weights of the fully connected layer can be dynamically generated from the representation relationship. The application of this technology greatly improves the fusion efficiency and module flexibility. By combining vision, text, and local relationships for spatial embedding learning, cross-modal retrieval can be achieved more accurately and efficiently.

[0135] Figure 5 The scheme structure diagram of the video text cross-modal retrieval method provided by the embodiment of the present invention is as follows: Figure 5 shown.

[0136] Considering the following problems of the retrieval method based on cross-modal feature vector matching: 1) For the video-text modal data encoding containing complex structures, after converting and mapping it to a single vector space, it is impossible to clearly represent the inherent structure and the correlation between cross-modalities, and it is impossible to make a direct and effective comparison; 2) Traditional video-text cross-modal retrieval methods are mostly based on image-text retrieval technology, mainly focusing on the basic feature matching of visual and text modalities. However, the visual modality has spatiotemporal characteristics that the image modality does not have, and traditional retrieval methods often ignore or pay insufficient attention to them, which limits the mining of video modal information and thus affects the extraction accuracy; 3) The extraction process pays too much attention to global features and ignores local details, resulting in low retrieval accuracy.

[0137] The cross-modal interactive retrieval method based on the attention mechanism has the following problems: 1) In the video-text dataset, the scale of video-text pairs is small, or although the number is sufficient, the feature diversity is lacking, which limits the generalization ability of the model; 2) It mainly adopts a single-scale, low-resolution attention mechanism interaction model between visual and text features. When extracting features, there is a lack of fine-grained feature interaction, resulting in low processing efficiency and insufficient accuracy; 3) Noise interference may be generated, which has a negative impact on the retrieval performance and increases the computational cost.

[0138] The aligned fusion retrieval method based on visual and text features has the following problems: 1) Insufficient utilization of fine-grained regional visual information leads to reduced matching accuracy; 2) In the fusion video-text cross-modality, the entanglement mechanism between different modalities is more complicated than the attention mechanism cross-modal interactive retrieval; 3) The computational cost of using graph convolutional networks (GCNs) to construct a complete spatiotemporal relationship graph is too high and the efficiency is low.

[0139] Therefore, an embodiment of the present invention proposes a method to solve the following technical problems: (1) How to realize the extraction of multi-level, fine-grained video and text features to obtain global and local modal representation information, perform cross-modal interactive learning on the multi-level representation information, and improve the retrieval accuracy; (2) How to fully explore the unique spatiotemporal characteristics of the video and improve the model's ability to understand dynamic content; (3) How to comprehensively utilize the advantages of multiple existing video-text retrieval methods to improve retrieval efficiency and accuracy, thereby optimizing the overall retrieval performance.

[0140] Please refer to Figure 5The method provided in the embodiment of the present invention can be mainly implemented through the following three modules, namely, a pre-training module based on ALPRO, a video-text feature extraction module based on multi-level analysis, and a spatiotemporal feature fusion module.

[0141] Among them, the ALPRO-based pre-training module is used to perform fine-grained video-text alignment on the original data in the target dataset, that is, the video data and its corresponding text data before feature extraction, to improve the efficiency of feature extraction. Correspondingly, the ALPRO-based pre-training module is used to extract the key frames of the video data; the key frames and their corresponding text data are input into the pre-training model to obtain the multimodal feature embedding vector.

[0142] The video-text feature extraction module based on multi-level analysis is used to accurately decompose the multimodal embedding vector obtained after processing by the pre-trained model into two dimensions of visual and text features based on multi-level analysis, based on the multi-level visual feature extraction module and the text feature extraction module, aiming to achieve the extraction of global-local, coarse-fine granularity representation. Correspondingly, the video-text feature extraction module based on multi-level analysis is used to extract features from the multimodal feature embedding vector to obtain video features and text features; wherein, the video features include video global features and video local features.

[0143] The spatiotemporal feature fusion module is used to perform effective fine-grained mining of video and text features through a custom spatiotemporal feature fusion module and joint embedding learning technology, thereby improving the model retrieval accuracy and efficiency while reducing the model complexity and computational cost.

[0144] For parts that are not described in detail in this embodiment, reference may be made to other related embodiments.

[0145] In summary, the embodiments of the present invention have the following advantages:

[0146] 1) Most existing works directly process the data set as input, or only perform basic preprocessing. However, data sets processed in this way often have problems of low quality and lack of feature diversity, which affects the generalization ability of the model and may limit the extraction efficiency and accuracy. In the embodiment of the present invention, the video data and text data in the original data are first deeply pre-trained before feature extraction. After obtaining more fine-grained modal data through modal alignment, retrieval work is performed, thereby significantly improving the efficiency and accuracy of cross-modal retrieval and reducing subsequent computing costs.

[0147] 2) Existing work does not fully explore the fine-grained features of vision and text, and is usually limited to the fine-grained feature extraction of a single modality, video or text. In addition, current methods mostly focus on hierarchical extraction of a single modality, and fail to achieve hierarchical interaction across modalities. The embodiments of the present invention combine fine-grained feature extraction and a hierarchical method based on semantic graphs, adopt hierarchical and fine-grained extraction methods for both video and text modalities, and realize interaction between modalities during the extraction process, effectively improving training efficiency and retrieval accuracy.

[0148] 3) Existing work does not dig deep enough into the spatiotemporal features of videos, or ignores effective integration with other modalities, thus limiting the accuracy of cross-modal retrieval. In addition, existing spatiotemporal fusion methods are inefficient and computationally expensive. The embodiments of the present invention make full use of the unique spatiotemporal characteristics of videos, and adopt an innovative spatiotemporal feature fusion method based on a cross-modal jump connection network and a residual Transformer. While reducing computational costs, it achieves efficient and deep integration of visual and textual modalities, comprehensively captures and utilizes information about video content, and improves the accuracy of cross-modal retrieval.

[0149] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0150] The following is an embodiment of the device of the present invention. For details not described in detail, reference may be made to the corresponding method embodiments described above.

[0151] Figure 6 The schematic diagram of the structure of the video text cross-modal retrieval device provided by the embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0152] like Figure 6 As shown, the video text cross-modal retrieval device 6 includes:

[0153] The acquisition module 61 is used to obtain the original data in the target data set; wherein the original data includes the video data and its corresponding text data;

[0154] A preprocessing module 62 is used to preprocess the original data to obtain a multimodal feature embedding vector;

[0155] An extraction module 63 is used to extract features from the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features;

[0156] A learning module 64 is used to respectively perform embedding learning on the video global features, the video local features and the text features to obtain a video embedding space, a video local relationship embedding space and a text embedding space;

[0157] The retrieval module 65 is used to perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space and the text embedding space.

[0158] In a possible implementation, the learning module 64 is specifically configured to:

[0159] The candidate region with the highest detection confidence in any frame of the local features of the video is used as the visual representation of the frame;

[0160] Based on visual representation, the data flow between entities contained in each frame of the local features of the video is processed;

[0161] Based on visual representation, the data flow between each frame of the local features of the video is processed;

[0162] According to the data flow between entities contained in each frame of the processed video local features and the data flow between each frame of the video local features, the video local relationship embedding space is obtained.

[0163] In a possible implementation, the learning module 64 is specifically configured to:

[0164] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0165] Inputting the text representation into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer; wherein the text representation is determined based on the text data;

[0166] The visual representation used to represent the data flow between entities contained in each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused first representation;

[0167] Passing the fused first representation through the feedforward network in the jump connection fusion module to obtain a first text representation with visual modality attention;

[0168] Inputting the first text representation with visual modality attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module;

[0169] After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the first multimodal representations are residually connected to determine the correlation between each frame in the local feature of the video and the local feature of the video as a whole.

[0170] In a possible implementation, the learning module 64 is specifically configured to:

[0171] For each skip connection fusion module in the cross-modal skip connection network, perform the following steps:

[0172] In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after the self-attention layer;

[0173] The visual representation used to represent each frame in the local features of the video is fused with the text features after the self-attention layer to obtain a fused second representation;

[0174] Passing the fused second representation through the feedforward network in the jump connection fusion module to obtain a second text representation with visual modality attention;

[0175] Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module;

[0176] After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the second multimodal representations are residually connected to obtain video local relationship information including spatiotemporal information.

[0177] In a possible implementation, the learning module 64 is specifically configured to:

[0178] The correlation between each frame in the local features of the video and the overall local features of the video, and the local relationship information of the video including spatiotemporal information are feature aggregated to obtain the local relationship embedding space of the video.

[0179] In a possible implementation, the retrieval module 65 is specifically configured to:

[0180] Mapping the video embedding space, the video local relationship embedding space and the text embedding space into the joint embedding space for joint embedding learning, and obtaining the visual-text cross-modal features after joint embedding learning;

[0181] The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output of the cross-modal encoder is used for video-text cross-modal retrieval.

[0182] In a possible implementation, the retrieval module 65 is specifically configured to:

[0183] In the joint embedding space, the global video-text and local relation-text are obtained based on the video embedding space, the video local relation embedding space and the text embedding space. The cosine similarity between the global video-text and the local relation-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

[0184] In a possible implementation, the preprocessing module 62 is specifically configured to:

[0185] Extract key frames of video data;

[0186] The key frames and their corresponding text data are input into the pre-trained model to obtain the multimodal feature embedding vector.

[0187] In a possible implementation, the extraction module 63 is specifically configured to:

[0188] The multimodal feature embedding vector is passed through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain the 2D appearance and 3D action features in the video data;

[0189] Combine 2D appearance and 3D motion features to obtain the global features of the video;

[0190] Extract the part of speech from the multimodal feature embedding vector, generate the corresponding semantic relationship graph based on the part of speech extraction result, and obtain the local features of the video based on the semantic relationship graph;

[0191] Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

[0192] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0193] Those of ordinary skill in the art will appreciate that the templates, units, and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0194] If the module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned video text cross-modal retrieval method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium.

[0195] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video text cross-modal retrieval method, characterized in that: include: Acquire original data in the target data set; wherein the original data includes video data and its corresponding text data; Preprocessing the raw data to obtain a multimodal feature embedding vector; Performing feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features; Embedding learning is performed on the video global features, the video local features and the text features respectively to obtain a video embedding space, a video local relationship embedding space and a text embedding space; Video-text cross-modal retrieval is performed based on the video embedding space, the video local relationship embedding space and the text embedding space.

2. The video text cross-modal retrieval method according to claim 1, characterized in that: The embedding learning of the local features of the video to obtain the local relationship embedding space of the video includes: Taking the candidate region with the highest detection confidence in any frame of the local features of the video as the visual representation of the frame; Based on the visual representation, processing the data flow between entities contained in each frame of the local feature of the video; Based on the visual representation, processing the data stream between each frame of the local feature of the video; The video local relationship embedding space is obtained according to the processed data flow between entities contained in each frame of the video local feature and the data flow between each frame of the video local feature.

3. The video text cross-modal retrieval method according to claim 2, characterized in that: The processing of the data stream between entities contained in each frame of the local feature of the video based on the visual representation includes: For each skip connection fusion module in the cross-modal skip connection network, perform the following steps: Inputting the text representation into a self-attention layer in any skip connection fusion module to obtain the text representation after passing through the self-attention layer; wherein the text representation is determined based on the text data; Fusing the visual representation of the data flow between entities contained in each frame in the local features of the video with the text features after passing through the self-attention layer to obtain a fused first representation; Passing the fused first representation through the feedforward network in the skip connection fusion module to obtain a first text representation with visual modality attention; Inputting the first text representation with visual modality attention and the visual representation of the data flow between entities contained in each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a first multimodal representation corresponding to the skip connection fusion module; After obtaining the first multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, each first multimodal representation is residually connected to determine the correlation relationship between each frame in the local feature of the video and the local feature of the video as a whole.

4. The video text cross-modal retrieval method according to claim 3 is characterized in that: The processing of the data stream between each frame of the local feature of the video based on the visual representation includes: For each skip connection fusion module in the cross-modal skip connection network, perform the following steps: In the frame-level space, the text representation is input into the self-attention layer in any skip connection fusion module to obtain the text representation after passing through the self-attention layer; Fusing the visual representation used to represent each frame in the local features of the video with the text features after passing through the self-attention layer to obtain a fused second representation; Passing the fused second representation through the feedforward network in the skip connection fusion module to obtain a second text representation with visual modality attention; Inputting the second text representation with visual modality attention and the visual representation used to represent each frame in the local features of the video into the cross-modal connection attention module in the skip connection fusion module to obtain a second multimodal representation corresponding to the skip connection fusion module; After obtaining the second multimodal representation corresponding to each skip connection fusion module in the cross-modal skip connection network, the second multimodal representations are residually connected to obtain video local relationship information including spatiotemporal information.

5. The video text cross-modal retrieval method according to claim 4, characterized in that: The method of obtaining a video local relationship embedding space based on the processed data flows between entities contained in each frame of the video local feature and the data flows between each frame of the video local feature comprises: The correlation relationship between each frame in the local features of the video and the local features of the video as a whole and the local relationship information of the video including the spatiotemporal information are feature aggregated to obtain a local relationship embedding space of the video.

6. The video text cross-modal retrieval method according to claim 1, characterized in that: The performing video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space and the text embedding space includes: Mapping the video embedding space, the video local relationship embedding space, and the text embedding space into a joint embedding space for joint embedding learning, and obtaining visual-text cross-modal features after joint embedding learning; The visual-text cross-modal features after joint embedding learning are passed through a cross-modal encoder, and the output result of the cross-modal encoder is used for video-text cross-modal retrieval.

7. The video text cross-modal retrieval method according to claim 6, characterized in that: The step of mapping the video embedding space, the video local relationship embedding space, and the text embedding space into a joint embedding space for joint embedding learning to obtain visual-text cross-modal features after joint embedding learning includes: In the joint embedding space, the global video-text and local relationship-text are obtained based on the video embedding space, the video local relationship embedding space and the text embedding space, the cosine similarity between the global video-text and the local relationship-text is calculated, the hinge-based ternary ranking loss is calculated, and the cosine similarity measure is optimized based on the balance of hyperparameters.

8. The video text cross-modal retrieval method according to claim 1, characterized in that: The preprocessing of the original data to obtain a multimodal feature embedding vector includes: Extracting key frames of the video data; The key frames and their corresponding text data are input into a pre-trained model to obtain a multimodal feature embedding vector.

9. The video text cross-modal retrieval method according to claim 1, characterized in that: The extracting features of the multimodal feature embedding vector to obtain video features and text features includes: Passing the multimodal feature embedding vector through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network respectively to obtain 2D appearance and 3D action features in the video data; Combining the 2D appearance and the 3D motion features to obtain a global video feature; Performing part-of-speech extraction on the multimodal feature embedding vector, generating a corresponding semantic relationship graph based on the part-of-speech extraction result, and obtaining a local feature of the video based on the semantic relationship graph; Hierarchical graph extraction and text part-of-speech extraction are used for the multimodal feature embedding vector, and text features are obtained based on the hierarchical graph extraction results and the text part-of-speech extraction results.

10. A video text cross-modal retrieval device, characterized in that: include: An acquisition module, used to obtain raw data in a target data set; wherein the raw data includes video data and its corresponding text data; A preprocessing module, used for preprocessing the raw data to obtain a multimodal feature embedding vector; An extraction module, used for performing feature extraction on the multimodal feature embedding vector to obtain video features and text features; wherein the video features include video global features and video local features; A learning module, used for respectively performing embedding learning on the video global features, the video local features and the text features to obtain a video embedding space, a video local relationship embedding space and a text embedding space; A retrieval module is used to perform video-text cross-modal retrieval based on the video embedding space, the video local relationship embedding space and the text embedding space.

Images