A method and apparatus for video retrieval using text

Abstract

The application provides a method and device for video retrieval using text, belonging to the technical field of computers, and the method comprises the following steps: inputting a key frame and a reference text feature into a preset deformation convolution network, using the reference text feature to guide the preset deformation convolution network to generate a dynamic receptive field, so as to extract a visual feature of the key frame; using a contrast learning method of momentum distillation to perform feature alignment on the visual feature and the reference text feature; inputting the aligned visual feature and the reference text feature into a modal fusion network, determining a target key frame with the highest similarity as a video retrieval result. The application redesigns an encoder of video data on a network structure, and uses text as a reference feature to guide the generation of a receptive field of a deformation convolution network, effectively solving the problems of too large model parameters, difficulty in loss function decline during training, and insufficient alignment of features before modal fusion, and giving consideration to the efficiency and accuracy of video retrieval using text.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for video retrieval using text. Background Technology

[0002] In recent years, due to the widespread adoption of the internet and mobile smart devices, as well as the rapid development of communication and social media technologies, a massive amount of online video data has been created and uploaded to the internet. In various scenarios, when people need to find the information they want from videos, they often need to spend a significant amount of time.

[0003] Convolutional neural networks (CNNs) are a hot topic in machine learning research and are frequently used in video information retrieval. Their advantage lies in transforming the original sample data layer by layer into a new feature space through multiple sequentially arranged convolutional and pooling layers, thus achieving automatic feature extraction. However, traditional CNNs indiscriminately scan the entire feature map from left to right and top to bottom, which undoubtedly wastes a lot of computational resources. To identify a keyframe, it is often only necessary to pay attention to a small, crucial area; this is the essence of attention mechanisms. However, attention-based structures often have a large number of parameters and lack the local modeling capabilities of CNNs, making it difficult to balance retrieval efficiency and accuracy.

[0004] Furthermore, patent application CN202210529260.2 discloses a method and system for retrieving infringing videos based on multi-visual expert knowledge distillation. Its main technical solution involves: acquiring initial features of the video through a pre-trained visual model; then encoding and training the initial features on a target dataset and performing comparative learning to obtain video feature representations with stronger task adaptability. Next, a knowledge fusion method is designed to fuse knowledge from multiple basic retrieval models. The fused knowledge is then transferred to the distillation retrieval model through knowledge distillation. The resulting distillation retrieval model, which integrates knowledge from multiple experts, exhibits higher computational efficiency and superior performance. However, the comparative learning method requires a large number of positive and negative samples, which is clearly insufficient in a single batch. Moreover, video-to-video fusion is a homomodal data fusion, and no alignment is required before fusion, severely limiting its application scenarios.

[0005] Therefore, there is an urgent need for a new technical solution to address the above problems, which can extract the information needed by users directly from lengthy videos using text data. Summary of the Invention

[0006] This invention provides a method for video retrieval using text, which addresses the shortcomings of existing technologies and balances the efficiency and accuracy of text-based video retrieval.

[0007] In a first aspect, the present invention provides a method for video retrieval using text, comprising: selecting keyframes from a target video at a preset frequency, and extracting reference text features from the text data using a text encoder; inputting the keyframes and reference text features into a preset deformable convolutional network, and using the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field to extract visual features of the keyframes; using a momentum distillation contrastive learning method to maintain a feature queue to provide a sufficient number of negative samples to align the visual features and the reference text features; inputting the aligned visual features and reference text features into a modality fusion network, ranking all selected keyframes by their similarity to the text data, and determining the target keyframe with the highest similarity as the video retrieval result.

[0008] According to the present invention, a method for video retrieval using text is provided, which extracts keyframes from a target video at a preset frequency, including: extracting all keyframes by selecting frames at fixed intervals and by selecting frames randomly; wherein, selecting frames at fixed intervals means selecting keyframes of the target video at fixed time intervals; and selecting frames randomly means randomly selecting two frames as keyframes within each time interval of selecting frames at fixed intervals.

[0009] According to the present invention, a method for video retrieval using text is provided, wherein the text encoder is the first six layers of a BERT network model, used to extract reference text features of the text data; the input of the first six layers of the BERT network model is the word vectors, positional encoding, and semantic encoding of the text data; the BERT network model includes a total of 12 layers.

[0010] According to the present invention, a method for video retrieval using text is provided, wherein the modality fusion network is the last six layers of a BERT network model, used to predict the similarity between keyframes and text data; the inputs of the last six layers of the BERT network model are aligned visual features and reference text features.

[0011] According to the present invention, a method for video retrieval using text is provided, wherein the preset deformable convolutional network is a ResNet-50 structure comprising 4 stages, and the third stage comprises 6 convolutional layers; wherein the 1st, 3rd and 5th convolutional layers in the third stage are replaced with deformable convolutions.

[0012] According to the present invention, a method for video retrieval using text is provided, which uses the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field in order to extract the visual features of key frames. The method includes: obtaining six text features of the text data generated by the first six layers of the BERT network model, and taking the text features corresponding to each two layers as a reference text feature; inputting the reference text features into the preset deformable convolutional network to generate a dynamic receptive field using deformable convolution.

[0013] According to the present invention, a method for video retrieval using text is provided, wherein the target video is an activity video in a student education scenario, and the text data is a descriptive text of the student activity.

[0014] Secondly, the present invention also provides an apparatus for video retrieval using text, comprising:

[0015] The acquisition module is used to select keyframes from the target video at a preset frequency and extract reference text features of the text data using a text encoder.

[0016] The deformable convolutional network guidance module is used to input keyframes and reference text features into a preset deformable convolutional network, and use the reference text features to guide the preset deformable convolutional network to generate dynamic receptive fields in order to extract the visual features of keyframes.

[0017] The feature alignment module is used to maintain a feature queue that provides a sufficient number of negative samples to align the visual features with the reference text features using a contrastive learning method based on momentum distillation.

[0018] The retrieval result output module is used to input the aligned visual features and reference text features into the modality fusion network, sort all selected keyframes by similarity to the text data, and determine the target keyframe with the highest similarity as the video retrieval result.

[0019] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for video retrieval using text as described above.

[0020] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for video retrieval using text as described above.

[0021] Compared with existing technical solutions, the present invention has the following obvious advantages and beneficial effects:

[0022] Multimodal data fusion often involves misaligned features. Existing methods simply use two separate feature extractors to obtain corresponding features before feature fusion, failing to fuse aligned features and thus unable to utilize mutual information between different modalities. This invention, however, utilizes contrastive learning and a method that guides visual features to form the dynamic receptive field of a deformable convolutional neural network. This fully aligns features between modalities before fusion, reducing the difficulty of the fusion stage and accelerating the convergence of the loss function during training. This results in a faster convergence speed and higher prediction accuracy with fewer parameters.

[0023] Most existing visual feature extractors are transformer-based, lacking local modeling capabilities, having a large number of parameters, and manually adding positional encoding to images, which is clearly insufficient. Video features contain both temporal and spatial information, while positional encoding only provides one-dimensional information and cannot accurately represent the positional relationships of images within keyframes. This invention employs a deformable convolutional network, which effectively solves the problem of local modeling capabilities, has fewer parameters, and since convolutional neural networks cover the entire image spatially, positional encoding is unnecessary as it inherently possesses positional information. Furthermore, deformable convolution can dynamically change the shape of the receptive field, achieving the same effect as attention mechanisms, and combining the advantages of both transformers and CNNs. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts illustrating the method for video retrieval using text provided by the present invention;

[0026] Figure 2 This is the second flowchart illustrating the method for video retrieval using text provided by the present invention;

[0027] Figure 3 This is a schematic diagram of the framework structure of the method for video retrieval using text provided by the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a pre-deformed convolutional network provided by the present invention;

[0029] Figure 5 This is a detailed diagram of the deformable convolutional network provided by the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] The following is combined Figures 1-6 This invention describes the method and apparatus for video retrieval using text, as provided in embodiments of the present invention.

[0034] Figure 1 This is one of the flowcharts illustrating the method for video retrieval using text provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps:

[0035] Step 101: Select keyframes from the target video according to a preset frequency, and use a text encoder to extract reference text features from the text data.

[0036] Optionally, the target video can be an activity video in a student educational setting, and the text data is descriptive text describing the student activity. This invention aims to retrieve video information from a target video using text data.

[0037] The text encoder can be the first six layers of a BERT network, used to extract reference text features from the text data; the inputs to the first six layers of the BERT network model are word vectors, positional codes, and semantic codes of the text data.

[0038] It should be noted that the BERT network in this invention is built on a transformer architecture, using only the transformer encoder part. Its overall framework consists of multiple layers of transformer encoders stacked together, but with the addition of mask language model and next sentence prediction training tasks. Each encoder layer consists of a multi-head attention layer and a feed-forward layer. The main function of each attention layer is to re-encode the target word based on its relevance to all words in the sentence.

[0039] Therefore, the calculation of each attention step includes three steps: calculating the relevance between words, normalizing the relevance, and obtaining the target word's encoding by weighted summation of the relevance and the encodings of all words. When calculating the relevance between words using attention, the input sequence vector is first linearly transformed using three weight matrices to generate three new sequence vectors: query, key, and value. The query vector of each word is then multiplied by the key vectors of all words in the sequence to obtain the relevance between words. This relevance is then normalized using the softmax() function. The normalized weights are then weighted and summed with the value to obtain a new encoding for each word.

[0040] As an optional embodiment, the present invention provides a method for video retrieval using text, which extracts keyframes from a target video at a preset frequency, including: extracting all keyframes by selecting frames at fixed intervals and by selecting frames randomly; wherein, selecting frames at fixed intervals means selecting keyframes of the target video at fixed time intervals; and selecting frames randomly means randomly selecting two frames as keyframes within each time interval of selecting frames at fixed intervals.

[0041] Specifically, video recording often involves capturing multiple images within one second. If all these images were used as input to a neural network, the network size would need to be matched accordingly, exceeding the machine's capacity. Therefore, this invention selects keyframes from the captured target video. The keyframes selected in this invention are divided into two parts, obtained through fixed-interval frame selection and random frame selection, respectively.

[0042] In a video, if we select p keyframes, the video's frame rate is q frames per second, and the total video length is s seconds, then the selected frames should correspond to positions with intervals of q / s. Using fixed-interval frame selection inevitably results in some frames with largely identical scenes, or the omission of moments where scenes change. Therefore, this invention incorporates random frame selection to compensate for the potential impact of fixed-interval frame selection. The random frame selection method involves randomly selecting two frames within each fixed-interval time interval as keyframes, and then overlaying these two keyframes with the keyframes selected using the fixed-interval method to create a single video keyframe.

[0043] Step 102: Input the keyframe and reference text features into a preset deformable convolutional network, and use the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field in order to extract the visual features of the keyframe.

[0044] Among them, Deformable Convolutional Networks (DCN) is an improvement based on ordinary convolutional neural networks. The receptive field of an ordinary convolutional neural network is in a convolutional block, and its size and shape are fixed. However, the receptive field of a DCN can change its shape according to the information of the image. The purpose of inputting text reference features into the DCN is to make it fully consider the information of the text when changing the shape of the receptive field.

[0045] As an optional embodiment, the preset deformable convolutional network in this invention is a ResNet-50 structure comprising 4 stages, and the third stage includes 6 convolutional layers; wherein, the 1st, 3rd and 5th convolutional layers in the third stage are replaced with deformable convolutions.

[0046] Specifically, the six text features obtained from the first six layers of BERT are used as reference text features for every two layers, and then used as input to the video encoder (deformable convolutional network). This allows the three ordinary convolutional blocks of the video encoder with a ResNet-50 backbone to be changed to deformable convolutions, taking into account the text information to change the receptive field, thus giving it a mechanism similar to self-attention.

[0047] Step 103: Using the momentum distillation contrastive learning method, maintain a feature queue to provide a sufficient number of negative samples to align the visual features with the reference text features.

[0048] This step relies on the contrastive learning method of Momentum Contrast (MOCO) in unsupervised visual representation learning. It maintains a feature queue, which has enough negative samples in the unsupervised case, so that the corresponding keyframe and reference text features are closer, and non-corresponding features are as far apart as possible, thereby learning a better feature distribution.

[0049] During feature extraction, contrastive learning requires a large number of negative samples. However, in the maintained queue, the distribution of the features originally output by the model changes drastically after one backpropagation as the parameters change, leading to a loss of consistency in the data within the continuous queue. This invention uses momentum updates to slowly update the parameters in the momentum model, thereby placing continuous and consistent features into the queue to optimize the effect of contrastive learning.

[0050] Step 104: Input the aligned visual features and reference text features into the modality fusion network, sort all selected keyframes by similarity to the text data, and determine the target keyframe with the highest similarity as the video retrieval result.

[0051] Optionally, the Bert network model in this invention includes a total of 12 layers, and the modality fusion network is the last six layers of the Bert network model, used to predict the similarity between keyframes and text data; the inputs of the last six layers of the Bert network model are aligned visual features and reference text features.

[0052] It should be noted that cross-modal fusion mainly uses the cross-attention mechanism, which is similar to the self-attention mechanism, except that the input comes from different sequences. Self-attention, on the other hand, uses the same sequence as the input, which means the inputs are different, but otherwise they are basically the same.

[0053] Specifically, the input to self-attention is a single embedding sequence. Cross-attention asymmetrically combines two embedding sequences of the same dimension, with one sequence used as the query Q input and the other as the key K and value V input. In short, the QKV of cross-attention is composed of two sequences, not a single one. This allows the model's final output to utilize both textual features and video information, achieving a multimodal fusion effect far superior to that of a single modality.

[0054] This invention provides a method for video retrieval using text. Before modality fusion, it uses contrastive learning, momentum distillation, and text-guided visual feature deformation convolution to obtain a video-text feature representation with stronger task adaptability. This fully aligns the features of the two modalities before fusion, thereby achieving better similarity prediction results for the input video and text features.

[0055] Figure 2 This is the second flowchart illustrating the method for video retrieval using text provided by the present invention. Figure 3This is a schematic diagram of the framework structure of the method for video retrieval using text provided by the present invention. See [link / reference]. Figure 2 and Figure 3 A brief explanation of the technical concept of this invention: This invention uses video and text to train a video retrieval model composed of a CNN and a transformer. This video retrieval model can select keyframes from collected videos of student educational scenarios, pass them through a deformable convolutional network, and allow reference text features to guide the deformable convolution to generate a dynamic receptive field, extracting visual features. This deformable convolutional network has two inputs: the original keyframes and reference text features from another modality. The reference text features are obtained using a BERT network model with layers 0 to 5. Furthermore, cross-modal attention is fused between the features of the two modalities (i.e., visual features and reference text features). During training, the output is the value of each loss function; during inference, the output is the data most relevant to the text / video. The specific implementation of this invention will be further described below based on the above technical concept.

[0056] Step 1: Data Preprocessing

[0057] The collected video data is placed in a folder, and the corresponding text data is placed in a text file to create a JSON file. The JSON file is a dictionary with the key "caption" storing the text data and "clip_name" storing the video name. Each time the model inputs data, it will look up the corresponding video name in the JSON file and find the video that corresponds to the text in the folder where the video is saved for training.

[0058] Step 2: Construct the network model

[0059] In this invention, a deformable convolutional network and a Bert network are constructed.

[0060] Because this invention requires the inclusion of text features, the deformable convolutional network structure has been redesigned to accept two input data. Figure 4 This is a schematic diagram of a pre-deformed convolutional network provided by the present invention, as shown below. Figure 4 As shown, this network is based on the ResNet-50 structure and consists of 4 stages. In the third stage, there are a total of 6 convolutional layers. Layers 1, 3, and 5 are replaced with deformable convolutions. The layer containing deformable convolutions is named the special bottleneck. Figure 5 This is a detailed diagram of the deformable convolutional network provided by the present invention, such as... Figure 5As shown, there are two paths during convolution. The upper path performs a dot product operation on video features (i.e., visual features) and text reference features to guide the receptive field offset generated by the video encoder (i.e., the deformable convolutional network). This offset is then used in the lower-level ordinary convolution, allowing the input feature map to change the shape of the receptive field within the convolution kernel, thereby achieving the purpose of deformable convolution. Optionally, this invention utilizes the forward() function to input the reference text features and video features together as parameters into the deformable convolutional network.

[0061] Optionally, the present invention can directly call the BERT model in the Transformer library to construct a BERT network model, and load the initialization parameters at the same time.

[0062] Step 3: Train the network model

[0063] Optionally, the network can be pre-trained on the publicly available video-text pair dataset “msrvtt_ret”. In downstream tasks, depending on the application, small self-made video-text datasets from different fields can be input into the network for fine-tuning, so that the model only needs a small portion of data to be applicable to different application fields.

[0064] Specifically, keyframes selected from the target video are input into a deformation convolutional network, and the corresponding text data from the target video is input into a text encoder. Figure 3 Bert 0-5 in the model. The features of each input are fed into the feature fusion network (...). Figure 3 In Bert 6-11, the prediction results of video text similarity are obtained. The data is propagated forward and backward through the network, and then the weight parameters in the network are updated by backpropagating the error corresponding to the one-hot vector.

[0065] For deformable convolutional networks: the input sample image size is 64*3*256*256, where 256*256 represents the image resolution, 64 is the batch size, and 3 represents the r, g, and b channels. (See attached diagram.) Figure 2 During the forward propagation process, starting from the second stage, each time the image passes through a stage, its height and width are halved, while the number of channels is doubled. Finally, a flatten function flattens the output to 2048 dimensions. After a linear mapping, each data point output by the backbone becomes a 768-dimensional feature, which is then reduced to 64*256 dimensions by another linear mapping.

[0066] For the BERT network: there are three inputs: word vectors, positional encodings, and semantic encodings. BERT segments each word and adds two special tokens: a "cls" token at the beginning of the sentence and a "sep" token separating sentences. The word vectors are input into the model for encoding. Each encoder performs residual block stacking and regularization operations. The output is a 64*25*768 feature set, where 64 is the batch size, 25 is the sequence length plus the special sequence, and 768 is the 768-dimensional word vector representing each word. The first word "cls" in the sequence of 25 words is used as the feature of the entire sentence. It also undergoes a linear mapping to project the dimension to 256.

[0067] It should be noted that, see Figure 3 The medium-momentum distillation method primarily aims to maintain two large and consistent video text queues within the model. Each time, the features output by the distillation model are stored in these queues, ensuring consistency among the features. The parameters in the distillation model are updated using the following formula:

[0068] θ k+1 ←mθ k +(1-)θ q

[0069] Where m∈[0,1] is a momentum coefficient, which is only the encoder parameter θ in the original model. q Only then will it be updated through backpropagation, and the encoder parameters θ of the current distillation model... k+1 Based on the previous θ k and the current θ q Indirect momentum update updates the parameters in the distillation model. The momentum update method in the formula makes θ k Ratio θ q The parameter updates are smoother, so the differences between encoder epochs can be small. In experiments, m is generally set to a large value (e.g., m = 0.95) to slowly update the encoder parameters in the distillation model.

[0070] Step 4: Output Results

[0071] The 256-dimensional features of video (keyframes) and text data are respectively input into the last six layers of BERT for cross-modal fusion. During the training phase, the matching between keyframes and text is predicted, thereby making full use of the information from both video and text modalities. Finally, the similarity ranking of all video keyframes relative to text is output.

[0072] Based on the above embodiments, as an optional embodiment, the important steps involved in the construction and training of the deformable convolutional network and BERT network used in the above embodiments of the present invention will be briefly described below.

[0073] It should be noted that, based on existing technology and the content of the above embodiments, the technical solution of the present invention is entirely feasible. The brief description below is only for the reader's easier understanding of the technical solution of the present invention and does not limit the scope of protection of the present invention.

[0074] First, the forward propagation stage

[0075] The convolution operations of the deformable convolutional network (DCN) during the forward propagation process are as follows:

[0076] On a 3x3 feature map, the coordinates of each small grid within the receptive field can be defined as:

[0077] R={(-1,-1),(-1,0),…,(0,1),(1,1)}

[0078] In the above formula, each coordinate represents the position of a grid point. During deformation convolution, the grid dynamically adjusts its coordinates based on the offset predicted by the feature map, thus allowing the receptive field of the feature map to dynamically adjust according to the offset. During the convolution operation, the convolution kernel and the offset feature map can be convolved, as shown in the following formula:

[0079]

[0080] Where, Δp n is the offset predicted by the network, w is the weight of the convolution kernel, and everything else is the same as a normal convolution.

[0081] The forward propagation process of BERT is shown below:

[0082] First, the text is segmented into words. Then, the index of each word in the vocabulary is input into the encoder. Each token is initialized with three weight matrices Q, K, and V. Q is multiplied by K for each token to obtain an attention score. Finally, each score is weighted and summed with V to obtain a new word vector representation. The specific formula is as follows:

[0083]

[0084] Where d kIt is a scaling factor, and softmax() is a function that normalizes the attention score. In this invention, Q and K are the same in the first six layers of BERT, and self-attention is calculated. After six such encoders, the text features are finally obtained. In the last six layers, it is the fusion model part, where Q is the video feature and V is the text feature, and cross-attention is calculated. Finally, the prediction of the similarity between the video and the text is output.

[0085] Second, the definition of the loss function.

[0086] Since the entire model has three main tasks: VTC, VTM, and MLM, there are also three loss functions, as shown in the appendix. Figure 3 VTC stands for Video-Text Contrast Learning Task, VTM stands for Video-Text Matching Modeling Task, and MLM stands for Masked Language Modeling Task.

[0087] (1) Video-text contrast loss function L vtc

[0088]

[0089]

[0090]

[0091] Where τ is a learnable temperature parameter, s(V,T) m ) represents the similarity between video and text features, p v2t (V) means that in the task of retrieving text from a video, the similarity between the video and each text is calculated, and then normalized using the softmax() function, p t2v (I) Similarly, the loss function L vtc In this context, H() represents the cross-entropy loss function, and y represents the ground truth label (0 / 1) of the similarity.

[0092] (2) Video text matching loss function L vtm

[0093] L vtm =H(y) vtm ,p vtm (V,T))

[0094] Where p vtm (V,T) represents the probability of keyframe and text matching in the model output, y vtm The meaning is the ground truth label indicating whether the keyframe and text match.

[0095] (3) Masked Language Model Lmlm

[0096] L mlm =H(y) msk ,p msk (V,T))

[0097] Where p msk (V,T) represents the model's predicted probability for the masked token. This is a built-in loss function in BERT used to train the model to generate text features.

[0098] Therefore, the overall loss function of the model is: L = L vtc +L vtm +L mlm +.

[0099] Third, an introduction to the Adam optimization algorithm.

[0100] Gradient descent is a commonly used algorithm in neural network training, but its fixed learning rate leads to some drawbacks. To address the inconsistency in convergence speed among different parameters caused by the fixed learning rate in gradient descent, this invention uses the Adam optimization algorithm, assigning an independent learning rate to each parameter. After calculating the gradient, parameters with larger gradients receive lower learning rates, and vice versa. Furthermore, to avoid independently calculating the gradient with each gradient update, which would cause the gradient direction to continuously change, Momentum incorporates the previous gradient value into the current gradient calculation, weighting the two together to obtain the updated value for the current batch of parameters. Adam combines these two considerations, adaptively setting the learning rate for each floating-point parameter while also taking into account past gradient history.

[0101] Understandably, the Adam optimization algorithm has been widely and successfully applied in deep learning, so we will not elaborate on the Adam optimization algorithm here.

[0102] The present invention also provides an apparatus for video retrieval using text, comprising:

[0103] The acquisition module is used to select keyframes from the target video at a preset frequency and extract reference text features of the text data using a text encoder.

[0104] The deformable convolutional network guidance module is used to input keyframes and reference text features into a preset deformable convolutional network, and use the reference text features to guide the preset deformable convolutional network to generate dynamic receptive fields in order to extract the visual features of keyframes.

[0105] The feature alignment module is used to maintain a feature queue that provides a sufficient number of negative samples to align the visual features with the reference text features using a contrastive learning method based on momentum distillation.

[0106] The retrieval result output module is used to input the aligned visual features and reference text features into the modality fusion network, sort all selected keyframes by similarity to the text data, and determine the target keyframe with the highest similarity as the video retrieval result.

[0107] It should be noted that the device for video retrieval using text provided in this embodiment of the invention can execute the method for video retrieval using text described in any of the above embodiments during specific operation, and will not be elaborated further in this embodiment.

[0108] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for video retrieval using text. This method includes: selecting keyframes from a target video at a preset frequency and extracting reference text features from the text data using a text encoder; inputting the keyframes and reference text features into a preset deformable convolutional network, using the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field to extract visual features of the keyframes; using a momentum distillation contrastive learning method to maintain a feature queue providing sufficient negative examples to align the visual features and the reference text features; inputting the aligned visual features and reference text features into a modality fusion network, ranking all selected keyframes by their similarity to the text data, and determining the target keyframe with the highest similarity as the video retrieval result.

[0109] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the method for video retrieval using text provided in the above embodiments, the method including: selecting keyframes from a target video at a preset frequency, and extracting reference text features of the text data using a text encoder; inputting the keyframes and reference text features into a preset deformable convolutional network, using the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field to extract visual features of the keyframes; using a momentum distillation contrastive learning method to maintain a feature queue to provide a sufficient number of negative samples to align the visual features and the reference text features; inputting the aligned visual features and reference text features into a modality fusion network, ranking all selected keyframes by similarity to the text data, and determining the target keyframe with the highest similarity as the video retrieval result.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the method for video retrieval using text provided in the above embodiments. The method includes: selecting keyframes from a target video at a preset frequency and extracting reference text features from the text data using a text encoder; inputting the keyframes and reference text features into a preset deformable convolutional network, using the reference text features to guide the preset deformable convolutional network to generate a dynamic receptive field to extract visual features of the keyframes; using a momentum distillation contrastive learning method to maintain a feature queue providing a sufficient number of negative samples to align the visual features and the reference text features; inputting the aligned visual features and reference text features into a modality fusion network, ranking all selected keyframes by similarity to the text data, and determining the target keyframe with the highest similarity as the video retrieval result.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for video retrieval using text, characterized in that, include: Keyframes are selected from the target video at a preset frequency, and reference text features of the text data are extracted using a text encoder. Keyframes and reference text features are input into a preset deformable convolutional network. The reference text features are used to guide the preset deformable convolutional network to generate a dynamic receptive field in order to extract the visual features of the keyframes. By employing a contrastive learning method based on momentum distillation, a feature queue is maintained to provide a sufficient number of negative samples for feature alignment between the visual features and the reference text features. The aligned visual features and reference text features are input into the modality fusion network. All selected keyframes are ranked according to their similarity to the text data, and the target keyframe with the highest similarity is determined as the video retrieval result. The text encoder consists of the first six layers of the BERT network model, used to extract reference text features from text data. The first six layers of the BERT network model take word vectors, positional codes, and semantic codes from text data as input; the BERT network model consists of a total of 12 layers. The preset deformable convolutional network is a ResNet-50 structure with 4 stages, and the third stage includes 6 convolutional layers; Specifically, the convolutions in layers 1, 3, and 5 of the third stage were changed to deformable convolutions.

2. The method for video retrieval using text according to claim 1, characterized in that, Extract keyframes from the target video at a preset frequency, including: Extract all keyframes using both fixed-interval and random frame selection methods; Fixed-interval frame selection refers to selecting keyframes of the target video at fixed time intervals; random frame selection refers to randomly selecting two frames as keyframes within each fixed-interval frame selection time interval.

3. The method for video retrieval using text according to claim 1, characterized in that, The modality fusion network consists of the last six layers of the Bert network model, used to predict the similarity between keyframes and text data. The inputs to the last six layers of the Bert network model are aligned visual features and reference text features.

4. The method for video retrieval using text according to claim 3, characterized in that, The reference text features are used to guide the pre-defined deformable convolutional network in generating a dynamic receptive field to extract visual features of keyframes, including: Obtain six text features from the text data generated by the first six layers of the BERT network model, and use the text features corresponding to every two layers as a reference text feature; The reference text features are input into the preset deformable convolutional network to generate a dynamic receptive field using deformable convolution.

5. The method for video retrieval using text according to claim 1, characterized in that, The target video is an activity video in a student education scenario, and the text data is a descriptive text of the student activities.

6. A device for video retrieval using text, characterized in that, A method for performing video retrieval using text as described in any one of claims 1 to 5, comprising: The acquisition module is used to select keyframes from the target video at a preset frequency and extract reference text features of the text data using a text encoder. The deformable convolutional network guidance module is used to input keyframes and reference text features into a preset deformable convolutional network, and use the reference text features to guide the preset deformable convolutional network to generate dynamic receptive fields in order to extract the visual features of keyframes. The feature alignment module is used to maintain a feature queue that provides a sufficient number of negative samples to align the visual features with the reference text features using a contrastive learning method based on momentum distillation. The retrieval result output module is used to input the aligned visual features and reference text features into the modality fusion network, sort all selected keyframes by similarity to the text data, and determine the target keyframe with the highest similarity as the video retrieval result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for video retrieval using text as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for video retrieval using text as described in any one of claims 1 to 5.

Images