A fine-grained video-text retrieval method based on contextual Transformer network

Abstract

A fine-grained video-text retrieval method based on a contextual Transformer network includes S1. obtaining global, object-level, and action-level features of video data; S2. encoding contextual information between the global, object-level, and action-level features; S3. inputting the global, noun, and verb features of the text into a graph reasoning model based on an attention mechanism to obtain fine-grained features of the text data; and S4. inputting the fine-grained features into a multi-level cross-modal matching module for similarity calculation. This method achieves improved performance on cross-modal video-text retrieval tasks.

Description

Technical Field

[0001] The present invention relates to the fields of video-text cross-modal retrieval and Transformer networks, and in particular, to a fine-grained video-text retrieval method based on contextual Transformer networks. Background Art

[0002] With the rise of numerous short video platforms, the application value and importance of cross-modal retrieval tasks have also increased. Video-text cross-modal retrieval aims to achieve superior performance in cross-modal video-text retrieval tasks by constructing a feature extraction model based on a neural network structure to obtain fine-grained features from both the video and text modalities. Video-text retrieval tasks involve inputting a video or text and matching it to data from another modality that corresponds to its content. The main research direction in cross-modal retrieval is to obtain more fine-grained modality representation features to improve retrieval performance.

[0003] Transformer networks have significantly advanced the field of natural language processing in recent years, and the recent introduction of the Vision Transformer network architecture demonstrates its ability to achieve impressive results on numerous computer vision tasks. Traditional video feature generation methods lack a hierarchical representation of semantic information, making it difficult to fine-grain video features. Consequently, there is room for improvement in the performance of cross-modal video-text retrieval tasks.

[0004] The difficulty in solving the above problems and defects lies in the following: To obtain more fine-grained video features, it is necessary not only to design a multi-level video feature extraction model to perform hierarchical feature extraction of semantic information within the video, but also to better preserve the semantic integrity of the video at the feature level, it is necessary to design a suitable relationship modeling network to model the contextual relationships between features at different levels. At the same time, for fine-grained video and text features, it is necessary to design a corresponding hierarchical cross-modal similarity calculation model to better match and align data between different modalities and further improve retrieval performance.

[0005] The significance of addressing these issues and deficiencies is that by constructing fine-grained video features, we can retain more semantic information at the feature level, reducing information loss during feature extraction, thereby achieving better performance in cross-modal video-text retrieval tasks. Furthermore, building a more fine-grained video-text retrieval model enables us to maintain strong retrieval performance even in more complex application scenarios. Summary of the Invention

[0006] The present invention provides a fine-grained video-text retrieval method based on the contextual Transformer network, which achieves better performance in cross-modal video-text retrieval tasks. The present invention also uses this network structure to encode contextual features between different levels, so that the representation features of the video modality contain more semantic information and more completely express the content of the original video, that is, to generate more fine-grained video features, and obtain a certain performance improvement in the final cross-modal retrieval task.

[0007] The technical solutions of the present invention are as follows:

[0008] The fine-grained video-text retrieval method based on a contextual Transformer network of the present invention comprises the following steps: S1. obtaining global hierarchical features, local object hierarchical features and local action hierarchical features of video data; S2. encoding contextual information between different hierarchical features of the global hierarchical features, local object hierarchical features and local action hierarchical features; S3. inputting the global features, noun features and verb features of the text into a graph reasoning model based on an attention mechanism to obtain fine-grained hierarchical features of the text data; and S4. inputting the fine-grained hierarchical features into a multi-level cross-modal matching module for similarity calculation.

[0009] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, in step S1, the global hierarchical features, local object hierarchical features and local action hierarchical features of the video data are obtained through the video hierarchical feature preliminary extraction framework.

[0010] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, in step S1, the global level features are obtained by a pre-trained global feature encoder, which is a combination of features of all image frames in the video, and is intended to describe the video content at a global level; the local object level features are obtained by a pre-trained local object feature encoder, and are intended to encode the object information appearing in the video; the local action level features are obtained by a pre-trained local action feature encoder, and are intended to encode the action information in the video.

[0011] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, in step S2, the global hierarchical features, local object hierarchical features and local action hierarchical features are input into the video hierarchical coding framework, and the contextual information between different hierarchical features is encoded through the context-based multi-level Transformer network.

[0012] Preferably, in the above-mentioned fine-grained video-text retrieval method based on contextual Transformer network, the context-based multi-level Transformer network is composed of a global context-based hierarchical Transformer network and a local context-based hierarchical Transformer network.

[0013] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, the hierarchical Transformer input based on the global context is the features of the local level and the global level. First, the local features are enhanced based on the self-attention mechanism based on the multi-head self-attention network, the regularization layer, and the feedforward neural network, and then the contextual interaction features between the global and local levels are constructed based on the multi-head cross-attention network.

[0014] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, the hierarchical Transformer input based on the local context is the features of two local levels, and the local features are enhanced based on the self-attention mechanism based on the multi-head self-attention network, the regularization layer, and the feedforward neural network, and then the context interaction features between the local and local levels are constructed based on the multi-head cross-attention network, and finally the global-object context features and action-object context features are obtained at the object level, and the global-action context features and object-action context features are obtained at the action level.

[0015] Preferably, in the above-mentioned fine-grained video-text retrieval method based on the contextual Transformer network, in step S4, cosine similarity is used as the metric function, the matching score is directly calculated at the global level, and the alignment operation between modalities is performed based on the cross-attention mechanism at the local level, and then the matching score is calculated. The final similarity score is obtained by weighted summation of the scores at different levels.

[0016] According to the technical solution of the present invention, the beneficial effects are as follows: the method of the present invention obtains more fine-grained video features, extracts features from different semantic levels, and models the interaction relationship between levels, retaining the semantic information of fine-grained video features to the greatest extent; at the same time, a multi-level cross-modal matching module based on the attention mechanism is constructed, which can align fine-grained features at different levels, thereby achieving better performance in cross-modal video-text retrieval tasks.

[0017] The Transformer network structure based on the cross-attention mechanism is often used to model model context information. The present invention also uses this network structure to encode context features between different levels, so that the representation features of the video modality contain more semantic information and more completely express the content of the original video, that is, to generate more fine-grained video features, and achieve a certain performance improvement in the final cross-modal retrieval task.

[0018] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below with reference to the accompanying drawings and through specific embodiments: BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0020] Figure 1 Flowchart of the fine-grained video-text retrieval method based on contextual Transformer network of the present invention;

[0021] Figure 2 This is the overall framework diagram of the fine-grained video-text retrieval method based on the contextual Transformer network of the present invention;

[0022] Figure 3 This is a framework diagram for the preliminary extraction of video-level features in the method of the present invention;

[0023] Figure 4 This is a structural diagram of the global context Transformer network in the present invention;

[0024] Figure 5 This is the structural diagram of the local context Transformer network in the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical methods and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the present invention.

[0026] This paper proposes that if the representational features of video modalities can contain more semantic information and more completely represent the original video content, that is, produce more fine-grained video features, it can achieve certain performance improvements in the final cross-modal retrieval task. Furthermore, for multi-layered fine-grained video features, it is necessary to design a corresponding hierarchical cross-modal matching module to align fine-grained features at different levels across modalities, further improving retrieval performance.

[0027] For video data, the semantic information inside the video is multi-layered, and there is an interactive relationship between different layers. In order to construct video features in a more fine-grained manner, the present invention first divides the semantic levels inside the video into: global event layer, local object layer, local action layer, and uses three different pre-training models to extract representation features of different levels for the next step of context information modeling. When the deep learning model can divide the original video features into different levels and perform global-local context information encoding, the model now has the ability to construct fine-grained video features, and the fine-grained features are conducive to improving the performance of the final cross-modal retrieval task. The experimental results on different data sets have proved the correctness of the present invention.

[0028] Based on the above description, the working principle of the present invention is: for the input video sample , use pre-trained models that extract features at different levels to extract global features respectively , local object features , local action features . Constructing a contextual Transformer network that encodes the global-local hierarchy , and a contextual Transformer network encoding the local-local level ; The encoded global context feature can be expressed as ,in Represents the contextual interaction features between the object level and the global level, represents the contextual interaction feature between the action level and the global level. Correspondingly, the local context feature is ; Finally, the global context features and local context features obtained after encoding are integrated into the original features of the corresponding level to further enhance the semantic richness of the video modality representation.

[0029] Combine Figures 1 to 5 The fine-grained video-text retrieval method based on the contextual Transformer network of the present invention comprises the following steps:

[0030] S1: Obtain the global level features, local object level features and local action level features of the video data. Specifically, first combine Figure 3The video-level feature preliminary extraction framework in

[15] is used to obtain global-level features, local object-level features, and local action-level features of the video data. Global-level features are obtained through a pre-trained global feature encoder and are a combination of features from all image frames within the video, aiming to describe the video content at a global level. Local object-level features are obtained through a pre-trained local object feature encoder and are designed to encode information about objects appearing in the video. Local action-level features are obtained through a pre-trained local action feature encoder and are designed to encode information about actions in the video.

[0031] S2: Encode the context information between the global level features, local object level features and local action level features. Specifically, the above features obtained in step S1 are input into Figure 2 The video level coding framework in [1] encodes the context information between different level features through a context-based multi-level Transformer network. It should be noted that Figure 2 The context-based multi-level Transformer network in Figure 4 The hierarchical Transformer based on global context and the hierarchical Transformer based on local context are combined. The hierarchical Transformer based on global context takes local and global features as input. It first enhances local features based on the self-attention mechanism using a multi-head self-attention network, regularization layers, and feedforward neural networks. Then, it constructs contextual interaction features between the global and local levels based on a multi-head cross-attention network. The hierarchical Transformer based on local context takes features from two local levels as input. It enhances local features based on the self-attention mechanism using a multi-head self-attention network, regularization layers, and feedforward neural networks. Then, it constructs contextual interaction features between the local and local levels based on a multi-head cross-attention network. Ultimately, global-object context features and action-object context features are obtained at the object level, and global-action context features and object-action context features are obtained at the action level. According to the feature fusion operation, the global-local context features and local-local context features of different levels are combined to output context-based local object-level features and context-based local action-level features.

[0032] S3: According to Figure 2 The text-level encoding framework inputs the global features, noun features, and verb features of the text into the graph reasoning model based on the attention mechanism (such as Figure 2 As shown, it involves global nodes, noun nodes, and verb nodes) to obtain fine-grained hierarchical features of text data. It should be noted that Figure 2 The text-level encoding framework of is based on the implementation method in the baseline model.

[0033] S4: Input the fine-grained hierarchical features obtained in step S3 (i.e., the fine-grained encoded video and text features) into the multi-level cross-modal matching module (e.g., Figure 2 As shown in Figure 2 As shown in the figure, cosine similarity is used as the metric function to directly calculate the matching score at the global level. At the local level, the cross-attention mechanism is used to align the modalities before calculating the matching score. The final similarity score is obtained by weighted summing the scores at different levels.

[0034] Table 1 Comparison of the results of the present invention and other algorithms on the MSR-VTT dataset

[0035]

[0036] Table 2 Comparison of the results of the present invention and other algorithms on the TGIF dataset

[0037]

[0038] Tables 1 and 2 respectively show the comparison of the results of this algorithm on the long video dataset MSR-VTT and the short video dataset TGIF with the baseline model algorithm in the cross-modal video-text retrieval task. If the correct matching result happens to fall into the retrieved retrieval results, it is called a successful retrieval. The number of successful retrievals divided by the total number of retrievals is the calculation method of the recall rate. The total recall rate is the sum of the recall rates when only the top 1, 5, and 10 retrieval results are taken as retrieval results during the retrieval process. Since we can obtain a ranked list of different matching results during the retrieval process, the median ranking is the median ranking of the correct results corresponding to each retrieval, and the mean ranking is the average ranking of the correct results. The benchmark model referenced by the present invention is HGR, and the model name of the present invention is CHT. It can be found from the table that CHT has a certain improvement over the baseline model HGR in many indicators such as the total recall rate, mean ranking, and median ranking for measuring cross-modal retrieval performance.

[0039] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the scope of protection of the present claims. Other embodiments and combinations of implementations of the present invention are also within the scope of protection of the present invention.

Claims

1. A fine-grained video-text retrieval method based on contextual Transformer network, characterized by: The following steps are involved: S1. Based on pre-trained models at different levels, obtain global-level features, local object-level features, and local action-level features of video data; S2. Encode contextual information between global-level features, local object-level features, and local action-level features; Inputting the global level features, the local object level features and the local action level features into a video level coding framework, and encoding context information between different level features via a context-based multi-level Transformer network; The context-based multi-level Transformer network is composed of a global context-based hierarchical Transformer network and a local context-based hierarchical Transformer network. The hierarchical Transformer based on global context takes local and global features as input. First, the local features are enhanced based on the self-attention mechanism based on a multi-head self-attention network, a regularization layer, and a feedforward neural network. Then, contextual interaction features between the global and local levels are constructed based on a multi-head cross-attention network. Specifically, through the hierarchical Transformer network based on global context, when the input is local object level and global level features, global-object context features can be obtained; when the input is local action level and global level features, global-action context features can be obtained. The local context-based hierarchical Transformer network inputs features of two local levels, and enhances the local features based on the self-attention mechanism based on a multi-head self-attention network, a regularization layer, and a feedforward neural network. Then, a multi-head cross-attention network is used to construct and generate contextual interaction features between local and local levels, obtaining action-object context features at the object level and object-action context features at the action level. According to the feature fusion operation, the global-object context features and the action-object context features are fused at the local object level to construct the context-based local object level features; The global-action context features and object-action context features are fused at the local action level to construct context-based local action level features; S3. Input the global features, noun features, and verb features of the text into the graph reasoning model based on the attention mechanism to obtain fine-grained hierarchical features of the text data; S4. Input the fine-grained hierarchical features into the multi-level cross-modal matching module to calculate feature similarity, and output the final similarity result between the video data and the text data.

2. The fine-grained video-text retrieval method based on contextual Transformer network according to claim 1 is characterized in that In step S1, the global level features, the local object level features and the local action level features of the video data are obtained through a preliminary video level feature extraction framework.

3. The fine-grained video-text retrieval method based on contextual Transformer network according to claim 1 is characterized in that In step S1, the global-level features are obtained through a pre-trained global feature encoder, which is a combination of features of all image frames in the video, and is intended to describe the video content at a global level; the local-level object features are obtained through a pre-trained local object feature encoder, and are intended to encode object information appearing in the video; the local-level action features are obtained through a pre-trained local action feature encoder, and are intended to encode action information in the video.

4. The fine-grained video-text retrieval method based on contextual Transformer network according to claim 1 is characterized in that In step S4, cosine similarity is used as the metric function to directly calculate the matching score at the global level, and the alignment operation between modalities is performed based on the cross-attention mechanism at the local level. Then, the matching score is calculated, and the final similarity score is obtained by weighted summation of the scores at different levels.

Images