Video text retrieval method based on pre-training model

Abstract

The present invention discloses a video text retrieval method based on a pre-trained model, which mainly solves the problem of low video text matching accuracy caused by data feature heterogeneity and redundant cascade in existing methods; the solution includes: 1) obtaining the frame sequence of the video data set and the text sequence generated by text annotation segmentation; 2) using the image and text pre-training model to collaboratively extract the features of the video and text; 3) mapping the video in a block cascade form, and applying the collaborative extraction to the aggregation stage of the video, using text representation to guide the aggregation of video frame-level features; 4) adding prior probabilities in cross directions and using decentralized constraint loss to complete model training; 5) using the trained video text matching model to obtain the final retrieval results. The present invention can effectively reduce the semantic gap between different modalities, fully explore the complex spatiotemporal information in video modal data, and improve cross-modal retrieval accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and further relates to video retrieval technology, specifically a video text retrieval method based on a pre-trained model, which can be used for semantic information mutual retrieval of large-scale video modality and natural language text modality data emerging on the Internet and social media, video topic detection, and content recommendation for video applications. Background Art

[0002] With the development of big data and 5G technology, multimedia data on the Internet has shown explosive growth, and at the same time has spawned many new retrieval needs, which has made traditional single-modal retrieval tasks gradually unable to meet diverse retrieval needs. Traditional retrieval methods mainly support concept-based retrieval for simple natural language text queries, which is ineffective for complex and long natural language text queries with complex semantics. Cross-modal retrieval, as a hot topic in the current field of multimedia research, can well meet users' search needs between different media data. In recent years, cross-modal retrieval methods based on shared subspaces have emerged, mapping video and natural language text modalities to a joint visual semantic shared space to calculate cross-modal semantic similarity as the basis for retrieval work.

[0003] In its patent application, "A Method for Cross-Modal Retrieval of Multiple Videos" (Patent Application No.: 201910453196.2, Publication No.: CN 110175266 A), Fudan University proposed a method for cross-modal retrieval of multiple videos based on deep neural networks and semantic feature relevance. This method, specifically designed for the complex task of cross-modal retrieval of long natural language texts, employs a bilateral self-attention method to extract features from videos relevant to the retrieval of natural language text. Specifically, a cross-modal detection model with a self-attention mechanism is used to initially select a small number of relevant video clips. Video clips are then further refined based on intra-modal and cross-modal similarity. Finally, a semantic ranking network is used to align the video clip sequence with the natural language text sentence sequence. However, this method has drawbacks: videos and natural language texts are data from different modalities. This heterogeneity makes it difficult to accurately represent the complex spatiotemporal semantic features of videos and natural language texts using a single deep neural network for single feature extraction. Consequently, it is difficult to measure the similarity between high-level features of the two, which affects retrieval accuracy. Hunan University proposed a method for extracting global and local object features of videos using multiple pre-trained models, and then finally achieving text-video retrieval through technical means such as affine transformation mapping in its patent application document with application number CN202111312233.1, titled "A cross-modal text-video retrieval method based on enhanced spatiotemporal relationships." Although this method improves retrieval accuracy to a certain extent by processing data of different modalities, the heterogeneity brought about by the use of separate feature extraction will result in the inability to accurately find the common embedding subspace between video and text even through mapping, resulting in poor matching between video and text due to modality differences, affecting retrieval accuracy.

[0004] Yang X, Dong J, Cao Y et al. proposed a tree-structured enhanced video natural language text retrieval method for complex natural language text queries in their paper "Tree-Augmented Cross-ModalEncoding for Complex-Query Video Retrieval" (International ACM SIGIRConference on Research and Development in Information Retrieval, 2020: 1339-1348). This method performs fine-grained encoding by jointly learning the language structure of the query natural language text and the temporal representation of the video. However, the spatial entity object of the video corresponds to the "noun" part of the natural language text, which is the key information for retrieval. The model of this method focuses on temporal modeling, which is difficult to capture the spatial object information at the video region level, thus affecting the accuracy of retrieval. Huaishao Luo, Lei Ji, Ming Zhong et al. proposed a cross-modal matching method for end-to-end video clip retrieval in their published paper "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval" (arXiv: Computer Vision and Pattern Recognition, 2104.08860 (2021)). The method extracts video and text information through a pre-trained model, transfers the knowledge of the image-text retrieval model to video language retrieval, and achieves SOTA results on multiple public datasets. Its shortcoming is that it aggregates the entire video without directly considering the text. Whether it is the mean pooling on the frame or the self-attention, it cannot achieve thorough collaborative feature extraction because it is not combined with the text expression, resulting in the fusion of a lot of redundant information into the video features, increasing the training burden, and generating a lot of undescribed visual information for the text encoding, affecting the retrieval results.

[0005] Most existing technologies use separate extraction of video and text features for mapping, which cannot effectively capture common features across modalities. Ignoring the correlation between features across different modalities results in features failing to effectively maintain intermodal relevance, creating a significant semantic gap between modalities and impacting the accuracy of subsequent cross-modal retrieval. Furthermore, existing technologies fail to capture the varying levels of importance represented by different frames when forming global video features, resulting in insufficiently expressive aggregated features. Therefore, finding a better approach to feature extraction and aggregation to address these issues is a pressing issue. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a video text retrieval method based on a pre-trained model to solve the problem of low video text matching accuracy caused by the heterogeneity of semantic features of data between different modalities and the redundant cascade of features within the same modality.

[0007] The idea behind this invention is to use the CLIP pre-trained model to collaboratively extract features from video and text, reducing the semantic gap between different modalities. Videos are then input into a Transformer in the form of patches for mapping, improving the model's fine-grained retrieval capabilities. This collaborative extraction is then applied to the video aggregation phase, using text representations to guide the aggregation of video frame-level features and enhance the expressiveness of the aggregated features. A new loss function is employed during the training phase to better achieve retrieval tasks. This invention can effectively reduce the semantic gap between different modalities, fully exploit the complex spatiotemporal information in video modality data, and improve cross-modal retrieval accuracy.

[0008] The present invention realizes the above-mentioned purpose scheme as follows:

[0009] The prior knowledge obtained from the image-text pre-training model is used to collaboratively extract video frame features and natural language text features, construct a common semantic subspace, aggregate video features using text knowledge, and train the video text retrieval model using cross-entropy loss and dispersion constraint loss functions. The implementation steps include:

[0010] (1) Processing data set:

[0011] Select the video dataset V to be trained and its corresponding text annotation T, and uniformly sample the video dataset according to the pre-specified sampling frequency through the video image generation tool to obtain the frame sequence of the video dataset in represents the jth frame of the i-th video in the video dataset; j = 1, 2, ..., N, N represents the number of frames in the video; i = 1, 2, ..., M, M represents the number of videos in the video set; at the same time, the text annotations are divided into words and stored in order to obtain the text sequence generated by T in Represents the text annotation T corresponding to the i-th video in the video dataset i The kth word after segmentation, k = 1, 2, ..., m, where m represents the total number of words in the text annotation;

[0012] (2) Collaborative extraction of video text features:

[0013] Extract images RGB features, i.e. red, green and blue color features, are obtained to obtain three sets of feature matrices; after extracting all the RGB features of each frame in all video frame sequences, each frame is cut into n×n blocks according to the given stride, and the position information and block features are concatenated and input into the pre-trained visual feature encoder VisualTransformer to obtain the frame-level representation sequence of the i-th video Sequence the text In the input text feature encoder Text Transformer, the features corresponding to the [EOS] instructions of the last layer of the text feature encoder are used as the representation of the input text sequence;

[0014] (3) Generate global feature representation of video text:

[0015] (3.1) Use a fully connected layer to represent the frame-level sequence of the i-th video Map it to the representation of the corresponding text sequence, and then perform normalization to obtain the i-th video mapped frame sequence and the text feature T of the i-th video i* ,in Represents the j-th frame image feature after the i-th video is mapped, and the similarity between the two is calculated

[0016] (3.2) Take the i-th video V i The sequence obtained by calculating the similarity between all frame image features and text features is enhanced through a bottom-up multi-head self-attention mechanism to enhance the subtle interactions within the video modality data and obtain the representation of the enhanced frame sequence.

[0017]

[0018] where ω1∈R 2D×D ,ω2∈R D×2D , D represents the dimension of each frame feature; tanh() represents the first activation function; Indicates video V i The j-th frame representation of

[0019] (3.3) Use LSTM network to construct video temporal information, randomly initialize LSTM parameter information, and obtain the i-th video V i The timing sequence V i* :

[0020]

[0021] in, Represents the i-th video V i The weight of the jth frame;

[0022] (3.4) According to the representation of the enhanced frame sequence and the time sequence V i* Get Video V i Global characteristics of At the same time, the text corresponding to the i-th video is annotated with T through the gate embedding module i Map to the public semantic embedding space to obtain text annotation T i The global feature Z i ;

[0023] (4) Through the video V i Global characteristics of and text annotation T i The global feature Z i The cosine similarity between the two is calculated to obtain the video text matching model;

[0024] (5) Training model:

[0025] (5.1) Calculate the prior probability of the cross direction of video and text, that is, calculate the prior probability of video features for the corresponding text features in all videos respectively and the prior probability of text features for all video features in all videos

[0026] (5.2) The prior probability of the cross direction and Introduced into the cross entropy loss, we get the matching loss from video to text. and text-to-video matching loss

[0027] (5.3) Using the dispersion constraint loss L c Constrain the video distribution:

[0028]

[0029] Among them, c i represents the mean of all video features in a batch, B represents the batch size, γ and μ represent the first and second adjustment parameters respectively;

[0030] (5.4) The total loss function L for model training is obtained:

[0031]

[0032] (5.5) Iteratively train the model parameters based on the total loss function L, and update the network's learning rate, neuron dropout rate, and loss function parameter values ​​in an arithmetic progression until L converges to a minimum, thereby obtaining a trained video-text matching model.

[0033] (6) Cross-modal mutual retrieval of video text:

[0034] The video text to be retrieved is input into the trained video text matching model, and the model output results are sorted to obtain the final retrieval results.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] First, since the present invention adopts a collaborative feature extraction method, by dividing the video into patches, it overcomes the difficulty in accurately modeling the information of each level of the video in the existing technology. Compared with traditional single-modal feature extraction, it can reduce the semantic gap between different modalities, fully explore the complex spatiotemporal information in the video modal data, and achieve the purpose of enriching the video representation capability.

[0037] Second, since the present invention adopts the method of text collaborative aggregation of video sequences, the redundant information in the video features is eliminated to a great extent through the participation of text in aggregation, and some frame features that are most relevant to the text content are retained, avoiding the video features in the training process from containing too much redundant information. It can accurately measure the similarity of high-order features of the same distribution of video and natural language text, thereby improving the accuracy of video natural language text retrieval.

[0038] Third, in view of the characteristics of video features such as high dimensionality and rich content, the present invention proposes to train the decentralized constraint loss together with the cross entropy loss to increase the similarity between video-text pairs and reduce the similarity between mismatched video-text pairs. At the same time, it increases the data difference within the video modality, thereby improving the accuracy of mutual retrieval between video and text, and effectively overcomes the problem of excessive similarity between video features and difficulty in distinguishing due to excessive concentration of features after mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the search process of the present invention;

[0040] Figure 2 It is an implementation flow chart of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] Example 1: Refer to the attached Figure 1 and Figure 2 The present invention proposes a video text retrieval method based on a pre-trained model. The method uses the prior knowledge obtained from the image-text pre-training model to collaboratively extract video frame features and natural language text features, construct a common semantic subspace, aggregate video features using text knowledge, and train the video text retrieval model using cross entropy loss and dispersion constraint loss functions. The method specifically includes the following steps:

[0043] Step 1, process the dataset:

[0044] Select the video dataset V to be trained and its corresponding text annotation T, and uniformly sample the video dataset according to the pre-specified sampling frequency through the video image generation tool to obtain the frame sequence of the video dataset Video image generation tools include ffmpeg, python and opencv, etc. In this embodiment, the sampling is preferably completed by the ffmpeg tool; represents the jth frame of the i-th video in the video dataset; j = 1, 2, ..., N, N represents the number of frames in the video; i = 1, 2, ..., M, M represents the number of videos in the video set; at the same time, the text annotations are divided into words and stored in order to obtain the text sequence generated by T in Represents the text annotation T corresponding to the i-th video in the video dataset i The k-th word after segmentation, k = 1, 2, ..., m, where m represents the total number of words contained in the text annotation.

[0045] Step 2: Collaboratively extract video text features:

[0046] Extract images RGB features, i.e. red, green and blue color features, are obtained to obtain three sets of feature matrices; after extracting all the RGB features of each frame in all video frame sequences, each frame is cut into n×n blocks according to the given stride, and the position information and block features are concatenated and input into the pre-trained visual feature encoder VisualTransformer to obtain the frame-level representation sequence of the i-th video Sequence the text In the input text feature encoder Text Transformer, the features corresponding to the [EOS] instructions of the last layer of the text feature encoder are used as the representation of the input text sequence.

[0047] Step 3: Generate global feature representation of video text:

[0048] (3.1) Use a fully connected layer to represent the frame-level sequence of the i-th video Map it to the representation of the corresponding text sequence, and then perform normalization to obtain the i-th video mapped frame sequence and the text feature T of the i-th video i* ,in Represents the j-th frame image feature after the i-th video is mapped, and the similarity between the two is calculated The calculation formula is as follows:

[0049]

[0050] Among them, |||| represents the modulo operation, (·) T Represents a transpose operation.

[0051] (3.2) Take the i-th video V i The sequence obtained by calculating the similarity between all frame image features and text features is enhanced through a bottom-up multi-head self-attention mechanism to enhance the subtle interactions within the video modality data and obtain the representation of the enhanced frame sequence.

[0052]

[0053] where ω1∈R 2D×D ,ω2∈R D×2D , D represents the dimension of each frame feature; Indicates video V i The j-th frame representation of ; tanh() represents the first activation function. Through the activation of the tanh function, the mapped sequence can now reflect the attention mechanism's selection tendency for the features of each frame of the video.

[0054] (3.3) Use LSTM network to construct video temporal information, randomly initialize LSTM parameter information, and obtain the i-th video V i The timing sequence V i* :

[0055]

[0056] in, Represents the i-th video V i The weight of the jth frame; the weight It is obtained through weight mapping. The specific expression is as follows:

[0057]

[0058] Wherein, e represents the natural index, l=1, 2, ..., N represents traversing each frame of the picture.

[0059] At this time, the weight value represents the importance of different video frames to the text, so as to focus on the aggregation of video frame features and ensure their aggregated expressiveness.

[0060] (3.4) According to the representation of the enhanced frame sequence and the time sequence V i* Get Video V i Global characteristics of That is, the weighted sum of the feature sequence with time series information, which is obtained according to the following formula:

[0061]

[0062] At the same time, the text corresponding to the i-th video is annotated with T through the gate embedding module i Map to the public semantic embedding space to obtain text annotation T i The global feature Z i ; The mapping process is carried out according to the following formula:

[0063] Z1=sigmoid(ωT i +b),

[0064] Where ω is a fully connected layer, b is the bias term, and sigmoid() represents the second activation function.

[0065] The obtained global feature Z i , specifically expressed as follows:

[0066]

[0067] Among them, ||·||2 represents the second normal form of the vector, Represents element-wise multiplication.

[0068] Step 4, by video V i Global characteristics of and text annotation T i The global feature Z i The cosine similarity between the two is calculated to obtain a video-text matching model.

[0069] Step 5: Train the model:

[0070] (5.1) Calculate the prior probability of the cross direction of video and text, that is, calculate the prior probability of video features for the corresponding text features in all videos respectively and the prior probability of text features for all video features in all videos The prior probabilities here are calculated according to the following formulas:

[0071]

[0072]

[0073] where η represents the first hyperparameter, the temperature coefficient, which is optimized directly as a logarithmically parameterized scalar during training.

[0074] (5.2) The prior probability of the cross direction and Introduced into the cross entropy loss, we get the matching loss from video to text. and text-to-video matching loss The details are as follows:

[0075]

[0076] Here, κ represents the second hyperparameter, which is set to 0.5 in this embodiment.

[0077] (5.3) In order to increase the distinction between different video features and reduce the degree of convergence of video features to the center, the dispersion constraint loss L is used. c Constrain the video distribution:

[0078]

[0079] Among them, c i represents the mean of all video features in a batch, B represents the batch size, γ and μ represent the first and second adjustment parameters respectively;

[0080] (5.4) The total loss function L for model training is obtained:

[0081]

[0082] (5.5) Iteratively train the model parameters according to the total loss function L, and update the network's learning rate, neuron drop rate, and loss function parameter values ​​in an arithmetic progression until L converges to a minimum, thereby obtaining a trained video-text matching model. In this embodiment, the Adam optimizer is preferably used to optimize the model parameters. The learning rate of the pre-trained model parameters is initialized to 1e-7, and the learning rates of other parameters are initialized to 1e-4. According to the total loss function L value, the model network is iteratively updated, and the network's learning rate, neuron drop rate, and loss function parameter values ​​are adjusted in an arithmetic progression within a reasonable range until the loss function converges to a minimum.

[0083] Step 6: Perform cross-modal mutual retrieval on video text:

[0084] The video text to be retrieved is input into the trained video text matching model, and the model output results are sorted to obtain the final retrieval results.

[0085] Example 2: The overall steps of this example are the same as those of Example 1, with specific settings made for the parameters, and step 2 of the present invention is further described in detail:

[0086] Collaborative extraction of video text features: extracting images RGB features, namely red, green and blue color features, to obtain three sets of feature matrices; at this time V iThe sequence shape is represented as frame×channel×width×height; after extracting all the RGB features of each frame in the video frame sequence, each frame of the sequence is cut into 14×14 patches according to stride=16, and then the position information and the patch are concatenated and input into the pre-trained Visual Transformer to fully extract the temporal spatial fine-grained features and interaction information, and then mapped to frame-level features to obtain the frame-level representation sequence of the i-th video Sequence the text The input text feature encoder, TextTransformer, uses a 12-layer, 512-wide pixel array and 8 attention heads. The maximum length of the input text sequence is 76, and the text sequence is surrounded by [SOS] and [EOS]. The features corresponding to the [EOS] instruction in the last layer of the text feature encoder are used as the representation of the input text sequence. The text and video representations in a batch are mapped sequentially to obtain a text-visual feature matrix. The video and text feature extractors are jointly trained through contrastive learning to predict N*N pairs of image and text data. This converts the image classification task into an image-text matching task, thereby narrowing the semantic gap between video and text features.

[0087] Example 3: The implementation steps of this example are the same as those of Example 1, and the method of the present invention is further described as a whole:

[0088] The present invention uses the latest image-text pre-training model CLIP as a collaborative feature extractor for video and text. As a feature extraction model trained on 400 million pairs of image-text data pairs, CLIP can well solve the heterogeneity problem between two different modalities of video and text. It is trained through a feature matrix composed of images and texts, and truly associates visual information with semantic information. In order to apply CLIP to the field of video text retrieval, the present invention adopts an average sampling method for video feature extraction, samples the video into 12-18 frames of RGB features, and then cuts the image features into 196 patches, which are input into the transformer together with the position code to obtain frame-level features with temporal information; the text feature extraction adopts the BERT model, cuts the sentence into words, and inputs them into BERT in sequence, and embeds the features represented by the [CLS] label as the global features of the sentence. Through the pre-training weights of CLIP, both video features and text features have good cross-modal performance capabilities. In order to aggregate the features of the frames into video-level features and enrich the aggregated expressiveness, the present invention adds text features to the aggregation process of video frame features, and uses text features to guide different frames of the video to be aggregated according to certain weights, thereby reducing the redundancy of features and improving the aggregation ability of video features. This feature extraction method fully models the spatial temporal information of video modal data with integrity and temporal sequence to obtain a comprehensive unified video representation, and retains the complex semantic information in the video for calculating the similarity of cross-modal video natural language text. The present invention uses a video natural language text common semantic embedding network to map two different modal data into a common semantic representation space. The common semantic representation space is designed to fit the underlying manifold structure of cross-modal data heterogeneity, so that the common semantic embedding network can effectively learn the common spatial representation of heterogeneous data of different modalities. In order to obtain better training results, the public semantic embedding network is made to map matching video-text pairs with high similarity and mismatching video-text pairs with low similarity. Based on the commonly used cross-entropy loss function, the present invention designs a new loss function, which introduces a priori probability weighting to ensure the diagonal dominance of the similarity matrix as much as possible. At the same time, a dispersion constraint loss is proposed to suppress the tendency of video features to converge to the center of the distribution mean, reflecting the differences between different video features. Under the constraints of these two loss functions, the public semantic embedding network of video and natural language text retains a high proportion of the cross-modal similarity semantics of video and natural language text features, and makes the positions of semantically similar video and natural language text feature mappings closer and semantically dissimilar video and natural language text feature mappings farther in the public semantic space. Using a semantic feature matching algorithm based on cosine similarity, the semantic association represented by the public space can be mined, the similarity between high-order features of video and natural language text can be measured, and the semantic matching similarity between video and natural language text can be calculated as the basis for cross-modal retrieval of video and natural language text.

[0089] The effects of the present invention will be further described below in conjunction with simulation experiments.

[0090] 1. Simulation conditions:

[0091] The hardware platform for the simulation experiment of the present invention is: NVIDIA GEFORCE RTX 3090 GPU.

[0092] The software platforms for the simulation experiments of the present invention are: Ubuntu 22 operating system and PyTorch 1.7.1.

[0093] The simulation data used in this paper is from the MSVD dataset collected by Chen and Dolan et al. in 2011. This dataset contains 1970 videos, each ranging from 1 to 62 seconds in length. The training set, validation set, and test set contain 1200, 100, and 670 videos, respectively. Each video has approximately 40 related English sentences, totaling 1970 video data points and 78,800 text data points.

[0094] 2. Simulation experiment content and simulation results analysis:

[0095] The simulation experiment used the present invention and five existing techniques to test the retrieval of the video corresponding to each natural language text in the test dataset and the natural language text corresponding to each video. Five networks were trained using the same training data, and the same test data was used as the retrieval samples to test the retrieval accuracy of each cross-modal information retrieval network.

[0096] In the simulation experiment, the five existing technologies used are as follows:

[0097] The existing technology HGR refers to the video natural language text retrieval method proposed by Chen S et al. in "Fine-grained video-text retrieval with hierarchical graph reasoning." (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.10638-10647, 2020), referred to as the fine-grained hierarchical graph reasoning HGR video natural language text retrieval method.

[0098] The prior art Frozen refers to the cross-modal information retrieval method proposed by Max Bain et al. in "Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval, international conference on computer vision, 2021", referred to as Frozen.

[0099] The existing technology CF-GNN refers to the cross-modal information retrieval method proposed by Wei Wang et al. in "Learning Coarse-to-Fine GraphNeural Networks for Video-Text Retrieval, IEEE TRANSACTIONS ON MULTIMEDIA 2021", referred to as CF-GNN.

[0100] The existing technology CLIP4Clip refers to the cross-modal information retrieval method proposed by Huaishao Luo et al. in "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.arXiv: Computer Vision and Pattern Recognition 2021", abbreviated as CLIP4Clip.

[0101] The existing technology CLIP2Video refers to the cross-modal information retrieval method proposed by Han Fang et al. in "CLIP2Video: Mastering Video-Text Retrieval via Image CLIP.arXiv: Computer Vision and Pattern Recognition, 2021", abbreviated as CLIP2Video.

[0102] In order to evaluate the effectiveness of the simulation experiment of the present invention, the recall rate, median rank, and mean rank of the ordered retrieval results of the present invention and the prior art were calculated, respectively, as shown in Table 1 below. Recall at K (R@K) represents the probability of correctly predicting the search term in the first K search results for the ordered search results; Median Rank (MedR) represents the median of the correctly predicted occurrence positions of the search term for the ordered search results; Mean Rank (MnR) represents the average number of correctly predicted occurrence positions of the search term for the ordered search results. A larger R@K value indicates a higher search accuracy, while smaller MedR and MnR values ​​indicate a higher search accuracy.

[0103] Table 1 Comparison of retrieval accuracy between the present invention and the prior art in simulation experiments

[0104]

[0105]

[0106] Table 1 shows the simulation results obtained by using the present invention and the prior art based on the three categories and five evaluation criteria of R@1, R@5, R@10, MedR and MnR. After comparative experiments with methods such as HGR and the present invention, a comparison table is obtained by statistically analyzing the retrieval results. The table is divided into two parts: retrieval accuracy evaluation of natural language text retrieval of video and video retrieval of natural language text.

[0107] It can be seen from Table 1 that the probabilities R@1, R@5, and R@10 of correctly predicting the items to be retrieved from the top 1, 5, and 10 retrieval results of natural language text retrieval of videos by the method of the present invention are 48.8, 78.6, and 87.6, respectively. The probabilities R@1, R@5, and R@10 of correctly predicting the items to be retrieved from the top 1, 5, and 10 retrieval results of natural language text retrieval of videos are 61.6, 88.5, and 94.0, respectively, which are much higher than the SOTA model fine-grained hierarchical graph reasoning HGR method in 2020, and higher than the SOTA models CLIP4Clip and CLIP2Video proposed in this field in 2021. In particular, the R@1 index of text retrieval video is improved by 430.4% compared with HGR, and it is also improved by 3.8% compared with the best performance model CLIP2Video in 21 years. The R@1 index of video retrieval text is improved by 310.7% compared with HGR, and it is improved by 4.9% compared with the best performance model CLIP2Video in 21 years. Experimental results show that this method can more accurately realize the mutual retrieval of videos and natural language texts, indicating that the retrieval accuracy of video natural language texts can be improved by collaborative feature extraction and aggregation through pre-trained models.

[0108] The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention.

[0109] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A video text retrieval method based on a pre-trained model, characterized in that: The prior knowledge obtained from the image-text pre-training model is used to collaboratively extract video frame features and natural language text features, construct a common semantic subspace, aggregate video features using text knowledge, and train the video text retrieval model using cross-entropy loss and dispersion constraint loss functions. The implementation steps include the following: (1) Processing data set: Select the video dataset V to be trained and its corresponding text annotation T, and uniformly sample the video dataset according to the pre-specified sampling frequency through the video image generation tool to obtain the frame sequence of the video dataset in represents the jth frame of the i-th video in the video dataset; j = 1, 2, ..., N, N represents the number of frames in the video; i = 1, 2, ..., M, M represents the number of videos in the video set; at the same time, the text annotations are divided into words and stored in order to obtain the text sequence generated by T in Represents the text annotation T corresponding to the i-th video in the video dataset i The kth word after segmentation, k = 1, 2, ..., m, where m represents the total number of words in the text annotation; (2) Collaborative extraction of video text features: Extract images RGB features, i.e. red, green and blue color features, are obtained to obtain three sets of feature matrices; after extracting all the RGB features of each frame in all video frame sequences, each frame is cut into n×n blocks according to the given stride, and the position information and block features are concatenated and input into the pre-trained visual feature encoder VisualTransformer to obtain the frame-level representation sequence of the i-th video Sequence the text In the input text feature encoder Text Transformer, the features corresponding to the [EOS] instructions of the last layer of the text feature encoder are used as the representation of the input text sequence; (3) Generate global feature representation of video text: (3.1) Use a fully connected layer to represent the frame-level sequence of the i-th video Map it to the representation of the corresponding text sequence, and then perform normalization to obtain the i-th video mapped frame sequence and the text features of the i-th video in Represents the j-th frame image feature after the i-th video is mapped, and the similarity between the two is calculated (3.2) Take the i-th video V i The sequence obtained by calculating the similarity between all frame image features and text features is enhanced through a bottom-up multi-head self-attention mechanism to enhance the subtle interactions within the video modality data and obtain the representation of the enhanced frame sequence. where ω1∈R 2D×D ,ω2∈R D×2D , D represents the dimension of each frame feature; tanh() represents the first activation function; Indicates video V i The j-th frame representation of (3.3) Use LSTM network to construct video temporal information, randomly initialize LSTM parameter information, and obtain the i-th video V i Time series in, Represents the i-th video V i The weight of the jth frame; (3.4) According to the representation and timing sequence of the enhanced frame sequence Get Video V i Global characteristics of At the same time, the text corresponding to the i-th video is annotated with T through the gate embedding module i Map to the public semantic embedding space to obtain text annotation T i The global feature Z i ; (4) Through the video V i Global characteristics of and text annotation T i The global feature Z i The cosine similarity between the two is calculated to obtain the video text matching model; (5) Training model: (5.1) Calculate the prior probability of the cross direction of video and text, that is, calculate the prior probability of video features for the corresponding text features in all videos respectively and the prior probability of text features for all video features in all videos (5.2) The prior probability of the cross direction and Introduced into the cross entropy loss, we get the matching loss from video to text. and text-to-video matching loss (5.3) Using the dispersion constraint loss L c Constrain the video distribution: Among them, c i represents the mean of all video features in a batch, B represents the batch size, γ and μ represent the first and second adjustment parameters respectively; (5.4) The total loss function L for model training is obtained: (5.5) Iteratively train the model parameters based on the total loss function L, and update the network's learning rate, neuron dropout rate, and loss function parameter values ​​in an arithmetic progression until L converges to a minimum, thereby obtaining a trained video-text matching model. (6) Cross-modal mutual retrieval of video text: The video text to be retrieved is input into the trained video text matching model, and the model output results are sorted to obtain the final retrieval results.

2. The method according to claim 1, wherein: The video image generation tools in step (1) include ffmpeg, python and opencv.

3. The method according to claim 1, wherein: Similarity in step (3.1) Calculated by the following formula: Among them, ||·|| represents the modulo operation, (·) T Represents a transpose operation.

4. The method according to claim 1, wherein: The i-th video V in step (3.3) i The weight of the jth frame It is obtained through weight mapping. The specific expression is as follows: Wherein, e represents the natural index, l=1, 2, ..., N represents traversing each frame of the picture.

5. The method according to claim 1, wherein: Global features in step (3.4) That is, the weighted sum of the feature sequence with time series information, which is obtained according to the following formula:

6. The method according to claim 1, wherein: In step (3.4), the text corresponding to the i-th video is annotated with T through the gate module GateEmbedding i Map to the public semantic embedding space to obtain text annotation T i The global feature Z i , the mapping process is performed according to the following formula: Z1=sigmoid(ωT i +b), Where ω is a fully connected layer, b is the bias term, and sigmoid() represents the second activation function.

7. The method according to claim 6, characterized in that: The global feature Z in step (3.4) i , the expression is as follows: Among them, ||·||2 represents the second normal form of the vector, Represents element-wise multiplication.

8. The method according to claim 1, wherein: The prior probability in step (5.1) is calculated according to the following formula: where η represents the first hyperparameter.

9. The method according to claim 1, wherein: The video-to-text matching loss in step (5.2) and text-to-video matching loss The details are as follows: Here, κ represents the second hyperparameter.

Images