Cross-modal video retrieval method and device based on dual-branch dynamic distillation learning

Abstract

The application discloses a cross-modal video retrieval method and device based on a double-branch dynamic distillation learning, which processes cross-modal video retrieval from a new angle, that is, extracts generalization knowledge from a large-scale visual language pre-training model and transfers the generalization knowledge to a task-specific network. The method is a double learning framework with dynamic knowledge distillation, and knowledge of the large-scale visual language model is used as a teacher to guide a student model. In the knowledge distillation process, an inheritance branch is designed to absorb the knowledge in the teacher model, and in order to avoid performance difference of the large-scale pre-training model due to domain gap, an exploration branch is designed to explore specific attributes of a downstream task. The network model provided by the application utilizes the double-branch structure, can deeply model the correlation between text and corresponding video, and thus effectively solves the text-to-video retrieval task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video cross-modal retrieval, and particularly relates to a cross-modal video retrieval method and device based on double-branch dynamic distillation learning. BACKGROUND

[0002] With the explosion of network videos, searching for interesting videos has become an indispensable activity in people's daily life. At the same time, text-to-video retrieval, i.e., retrieving videos from a large number of unlabeled videos by text query, has recently attracted more and more attention. Given a text query, the task of cross-modal video retrieval aims to retrieve relevant videos with the query from a set of video clips. The common solution is to first encode the video and the text query, and then map them to a common embedding space where cross-modal similarity can be directly measured. Current work mainly focuses on video encoding, text encoding and cross-modal similarity learning.

[0003] For various cross-modal tasks such as text-image retrieval, visual question answering, etc., more and more large-scale pre-trained visual and language models are used, such as contrastive language-image pre-training (CLIP), and ideal performance is achieved. For the text-to-video retrieval task, current work mainly focuses on the learning of the temporal aggregation layer above the features, because videos are mainly composed of image sequences, and pre-trained models are only trained on image-text pairs. Therefore, directly migrating large-scale pre-trained models trained mainly on image-text to the video-text field may result in a huge performance gap on the dataset. Therefore, how to effectively transfer the knowledge of the language-image pre-trained large model to the cross-modal video retrieval task is still a pending problem.

[0004] To this end, the present application proposes a double learning framework with dynamic knowledge distillation to transfer the knowledge of large models to cross-modal tasks. Specifically, an effective teacher-student network is developed, in which the CLIP model is used as the teacher, and a double-branch student model is designed to acquire knowledge. The reason for introducing two student branches is that, due to the complexity of the dataset, CLIP may encounter a domain gap problem, therefore, a legacy student branch is introduced to directly absorb the beneficial knowledge of the teacher model in a specific domain, while the other exploration student branch is used to only explore the task-specific properties of the training data. In addition, for humans, first learn from the teacher, and once their initial cognition is formed, they will gradually evolve through self-life learning. Therefore, a dynamic knowledge distillation strategy is designed, i.e., the legacy branch occupies the primary position at the beginning, while the exploration branch gradually becomes more prominent during the training process. SUMMARY

[0005] The application aims to solve the performance difference problem of pre-training large models when applied to downstream tasks, and proposes a cross-modal video retrieval method and device based on double-branch dynamic distillation learning.

[0006] The purpose of the application is achieved by the following technical solutions:

[0007] According to the first aspect of the present application, a cross-modal video retrieval method based on double-branch dynamic distillation learning is provided, which comprises the following steps:

[0008] (1) Feature pre-extraction is performed on the query text and the video to be retrieved to obtain the text features and video features of the teacher model;

[0009] (2) The text features and video features of the teacher model obtained in step (1) are subjected to similarity calculation to obtain a semantic similarity distribution as a guide for the inheritance branch of the student model;

[0010] (3) Feature pre-extraction is performed on the query text and the video to be retrieved to obtain the initial text features and initial video features of the student model;

[0011] (4) On the inheritance branch of the student model, the initial text features obtained in step (3) are encoded to obtain text feature representations;

[0012] (5) On the inheritance branch of the student model, the initial video features obtained in step (3) are subjected to feature encoding to obtain video feature representations;

[0013] (6) The video feature representations obtained in step (5) and the text feature representations obtained in step (4) are subjected to similarity calculation to obtain the similarity of each frame of the video and the text of the inheritance branch, and the maximum value is taken as the similarity of the video and the text of the inheritance branch;

[0014] (7) The similarity of each frame of the video and the text obtained in step (6) and the similarity of the video and the text are used to train the inheritance branch through a common space learning algorithm and a dynamic distillation algorithm, so that the inheritance branch can absorb beneficial knowledge from the teacher model;

[0015] (8) The exploration branch and the inheritance branch of the student model share the same network architecture, and after the same feature encoding and similarity calculation as steps (4), (5) and (6) of the inheritance branch, the similarity of each frame of the video and the text and the similarity of the video and the text of the exploration branch are obtained;

[0016] (9) The similarity of each frame of the video and the text obtained in step (8) and the similarity of the video and the text are used to train the exploration branch through a common space learning algorithm, so as to learn the knowledge on the cross-modal retrieval task specific dataset;

[0017] (10) By jointly training the dual-branch dynamic distillation learning network model including the inheritance branch and the exploration branch through steps (7) and (9), the text and video are input into the trained dual-branch dynamic distillation network model to realize cross-modal retrieval from text to its related video.

[0018] Furthermore, in step (1), different pre-trained models are used to extract text features and video features from the teacher model, including:

[0019] (1-1) Use the pre-trained large-scale text feature extractor CLIP model to extract text features of the teacher model;

[0020] (1-2) Divide the video into a sequence of video frames and use the pre-trained large-scale image feature extractor CLIP model to extract features from the video frame sequence to obtain the teacher model video features.

[0021] Further, in step (2), the video frame-by-frame feature sequence obtained in step (1-2) and the text features obtained in step (1-1) are input into the teacher model. For each video frame-by-frame feature and text feature, the cosine similarity is calculated to obtain the semantic similarity distribution as a guide for the inheritance branch of the student model.

[0022] Furthermore, in step (3), different pre-trained models are used to extract text features and video features from the student model, including:

[0023] (3-1) Use the pre-trained large text feature extractor RoBERTa model to extract the feature vector set of words and obtain the initial text features of the student model;

[0024] (3-2) Divide the video into a sequence of video frames and use the 2D deep convolutional network ResNet152 pre-trained on ImageNet to extract features from the video frame sequence to obtain the initial video features of the student model.

[0025] Furthermore, the method for encoding the initial text features obtained in step (3) includes: the inheritance branch uses a fully connected layer and a ReLU activation function to reduce the dimensionality of the input initial text features, and then performs position embedding encoding on the dimensionality-reduced text features before inputting them into the transformer to capture their context information, thereby obtaining the text feature representation of the inheritance branch.

[0026] The method for encoding the initial video features obtained in step (3) includes: the inheritance branch uses a full connection layer and a ReLu activation function to reduce the dimension of the input initial video features, and the reduced video features are input into a transformer after position embedding coding to capture their context information, and a frame scale feature sequence of the inheritance branch video is obtained through a full connection layer, that is, the video feature representation of the inheritance branch.

[0027] Further, the method for training the inheritance branch in step (7) is specifically:

[0028] The inheritance branch is trained in a dynamic distillation manner; the similarity between the video and the text is learned through the ternary sorting loss and the contrast learning loss; as the training goes deeper, the inheritance branch learns more and more knowledge from the teacher model, dynamically adjusts the weight of the distillation loss, and absorbs the beneficial knowledge from the teacher model through the dynamic distillation loss.

[0029] Further, the total loss for training the inheritance branch is:

[0030]

[0031] Among them, represents the ternary sorting loss of the inheritance branch, represents the contrast learning loss of the inheritance branch, represents the dynamic distillation loss, λ1 is a preset weight coefficient, w is an online adjusted hyperparameter during model training, w=w0g(t), wherein w0 is an initial weight, t represents the tth period during training, g(t) is a decay function, including exponential decay, linear decay, Sigmoid decay; the formula of the distillation loss is:

[0032]

[0033] Among them, k represents the number of video frames, represents the similarity between each frame of the video and the text in the student model inheritance branch, represents the similarity between each frame of the video and the text in the teacher model.

[0034] Further, the method for training the exploration branch in step (9) is specifically: the exploration branch is trained by using a public space learning algorithm, and the specific properties of the downstream task and the similarity between the video and the text are explored through the ternary sorting loss and the contrast learning loss.

[0035] Further, in step (10), the loss of the double-branch dynamic distillation learning network is:

[0036]

[0037] wherein, represents the loss of the inheritance branch, represents the loss of the exploration branch.

[0038] According to a second aspect of the present specification, a cross-modal video retrieval device based on double-branch dynamic distillation learning is provided, comprising a memory and one or more processors, the memory has stored executable code, and the processor executes the executable code to realize the cross-modal video retrieval method based on double-branch dynamic distillation learning as described in the first aspect.

[0039] The present application has the beneficial effects that: the present application provides a cross-modal video retrieval method and device based on double-branch dynamic distillation learning. In the method, the video is represented as a frame-scale feature, which is input into a double-branch dynamic distillation learning network. The double-branch dynamic distillation learning network includes an inheritance branch and an exploration branch. The inheritance branch calculates the similarity between each frame of the video and the query text, and takes the maximum value as the similarity of the video and the query text. The exploration branch performs the same operation as the inheritance branch. Finally, the similarity of the video and the query text is jointly measured using the inheritance branch similarity and the exploration branch similarity. The method of the present application utilizes the idea of double-branch dynamic distillation, and dynamic distillation helps to deal with the problem that the distillation effect is not good due to the performance of different data sets. Due to the field gap, the teacher model cannot always perform well on various data, so when the teacher model performs poorly, the inheritance student branch may be prone to errors. Therefore, the present application designs another student branch, called the exploration branch, which only learns the data-specific properties of the training set without any guidance from the teacher model. By jointly training the two branches, the advantages of the teacher model on well-performing data can be effectively utilized, while the negative impact of the teacher model's performance decline on certain data is reduced. Unlike the inheritance branch, which updates its similarity distribution by referring to the teacher's knowledge, the exploration branch is designed to learn data-specific knowledge directly from the live training data. And the inheritance branch and the exploration branch in the model network of the present application have a mutually complementary effect on the video representation. In the case that the inheritance branch may have deficiencies in understanding the video, the exploration branch can help the inheritance branch to supplement the missing information. The network model proposed in the present application utilizes the above branch structure, which can deeply model the correlation between the text and the corresponding video, thereby effectively solving the text-to-video retrieval task. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A cross-modal video retrieval method based on double-branch dynamic distillation learning provided by the present application is shown in the figure.

[0041] Figure 2A cross-modal video retrieval device structure schematic diagram based on double-branch dynamic distillation learning is provided. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0043] For the task of cross-modal video retrieval oriented to practical application cases, the present application proposes a cross-modal video retrieval method based on double-branch dynamic distillation learning, which can effectively mine the relevance of videos and their corresponding texts. In this method, the video is represented as a frame-scale feature. After converting the video into a frame-scale representation, it is input into the double-branch dynamic distillation network. The inheritance branch of the student model mainly absorbs knowledge from the large model, considering that due to the domain gap, the large model may perform poorly, and a student model exploration branch is further developed to take advantage of task-specific information. By jointly training the above two branches in a dual learning manner, the model can capture task-specific attributes while selectively obtaining appropriate knowledge from the teacher model. The double-branch dynamic distillation learning network includes an inheritance branch and an exploration branch, which learn video representations in an inheritance and exploration manner respectively, and the two similarity learning branches are complementary in information. The inheritance branch and the exploration branch simultaneously calculate the similarity of the video and the text. Finally, the similarity of the video and the query text is jointly measured using the inheritance branch similarity and the exploration branch similarity.

[0044] As Figure 1 shown, the cross-modal video retrieval method based on double-branch dynamic distillation learning provided by the embodiment of the present application has the following specific steps:

[0045] (1) Use different feature extraction methods to extract features of the two modalities of the teacher model video and text.

[0046] (1-1) Given a sentence composed of n q words, use the contrastive language-image pre-training model (CLIPText Encoder model) to extract the feature vector q t as the initial feature of the text.

[0047] (1-2) Given a video, first specify a video frame sequence with an interval of 1.5 seconds, obtaining k video frames. Use the contrastive language-image pre-training model (CLIP Image Encoder model pre-trained on WebImageText) to extract the features of the video frame sequence, as the initial feature vector set of the video, represents the initial feature vector of the kth video frame.

[0048] (2) The video frame-by-frame feature sequence F obtained in step (1-2) is input into the student model to calculate the similarity between F and the text feature representation q obtained in step (1-1). t t t t t

[0049]

[0050] (3) The features of the student model video and text of the two modalities are extracted using different feature extraction methods.

[0051] (3-1) Given a sentence composed of n q words, the pre-trained RoBERTa model is used to extract the feature vector set of the words. as the initial feature of the text, represents the feature vector of the nth q word.

[0052] (3-2) Given a video, first specify the video frame sequence with an interval of 1.5 seconds to obtain k video frames. The 2D deep convolutional network ResNet152 pre-trained on ImageNet is used for feature extraction of the video frame sequence, V = {v1, v2,..., vk} as the initial feature vector set of the video, vk represents the initial feature vector of the kth video frame. k k

[0053] Through the above feature extraction steps, the initial features of the video and the text are obtained, and then a deeper feature representation is needed.

[0054] (4) The initial text feature obtained in step (3) is input into the sentence feature representation encoding module of the inheritance branch of the student model for encoding, and the specific steps are as follows:

[0055] (4-1) The initial text feature Q is reduced in dimension (from 768 dimensions to 384 dimensions) through a fully connected layer (FC) and a ReLu activation function, and the reduced text feature is positionally embedded (PE).

[0056] ​​​​​​​​(4-2) The text features obtained in step (4-1) are input into the transformer to capture its contextual information. In the transformer, the text features pass through a multi-head self-attention layer and a feedforward layer in sequence. Both of these encoding layers are accompanied by residual connections and layer normalization operations, i.e.:

[0057] Q′=Transformer(ReLu(FC(Q))+PE)

[0058] In the above formula, Transformer represents the standard transformer module, and PE represents the position embedding coding module.

[0059] (4-3) The text features obtained in step (4-2) are processed through an attention mechanism. The features are aggregated into a single text feature vector. The weight vector w is multiplied by Q′ and passed through a softmax layer to obtain n. q Weight α q The calculated weights are used to perform a weighted sum on the text features Q′ to obtain the final text feature representation q of the inheritance branch. s ,Right now:

[0060]

[0061] Where q i Let be the i-th feature value in the text feature vector Q′.

[0062] (5) Input the initial video features obtained in step (3) into the video feature representation encoding module of the inheritance branch of the student model for encoding. The specific steps are as follows:

[0063] (5-1) The initial video features V are reduced in dimensionality (from 3072 dimensions to 384 dimensions) by a fully connected layer (FC) and the ReLU activation function, and the reduced video features are then encoded by position embedding (PE).

[0064] (5-2) Input the video features obtained in step (5-1) into a transformer with the same structure as in step (4-2) to capture its contextual information, and obtain the frame-scale feature sequence of the inherited branch video through a fully connected layer (FC), that is, the video feature representation F of the inherited branch. s The formula is as follows:

[0065] F s =FC(Transformer(ReLu(FC(V))+PE))

[0066] The inheritance branch and the exploration branch share the same network architecture in the above steps, and the structures of the full connection layer (FC), the position embedding encoding (PE), and the transformer module used by the sentence feature representation encoding module and the video feature representation encoding module are all the same, but only the full connection layer (FC) shares parameters, and the rest do not share parameters. After the exploration branch is encoded through the same steps as the inheritance branch, the text feature representation of the exploration branch is obtained and the video feature representation of the exploration branch

[0067] Through the above steps, the inheritance branch meaning text feature representation is obtained the text feature representation of the exploration branch the video feature representation of the inheritance branch and the video feature representation of the exploration branch Large-scale pre-trained visual and language models are increasingly used, such as the contrastive language-image pre-training model (CLIP), however, CLIP features can cause huge performance differences on different datasets. Therefore, the present application constructs a double-branch dynamic distillation learning network model including an inheritance branch and an exploration branch; through the method of dynamic distillation, the knowledge of an external large model is introduced to help measure the similarity between text and corresponding video. Through the method of double branches, task-specific properties are captured. This calculation method is based on an assumption: humans first learn from teachers, and once they have formed their own preliminary cognition, they will gradually evolve and learn from their own life. Therefore, a dynamic knowledge distillation strategy is designed, that is, the inheritance branch occupies the primary position at the beginning, and the exploration branch gradually becomes more prominent during the training process. First, the similarity of the inheritance branch video and text is detected, and then the similarity of the exploration branch video and text is detected. The similarity of the inheritance branch and the exploration branch video and text is considered as the final similarity.

[0068] (6-1) Dot product is performed on the inheritance branch text feature and the inheritance branch video feature sequence obtained in step (5) to calculate the cosine similarity of the text feature representation q s and the video feature sequence F s as the similarity of each frame of text and video of the inheritance branch, that is, the similarity set C s :

[0069]

[0070] (6-2) The maximum value of the similarity of each frame of text and video of the inheritance branch in step (6-1) is taken as the similarity S I (Q, V) of the inheritance branch video and text, that is:

[0071]

[0072] The inheritance branch text and video frame-by-frame similarity set C is obtained through the above steps s and the video and text similarity S I (Q, V). Then the dynamic distillation and common space learning algorithm are used to absorb the beneficial knowledge of the teacher model and the correlation between the text and video modalities. The specific steps are as follows:

[0073] (7) In multi-instance learning, there is a concept of bag. The bag can be simply understood as a set composed of multiple samples. If one or more instances in the bag are positive samples, the bag is a positive sample; otherwise, the bag is a negative sample. Based on the above definition, the triplet ranking loss and contrastive learning loss widely used in retrieval tasks are used to jointly constrain the inheritance branch and the exploration branch. The formula of the triplet ranking loss is as follows:

[0074]

[0075] Where n is the number of samples, m is a boundary constant, the value is 0.2, and S(·) is a similarity function. q - and v - are the text negative sample of the video v and the video negative sample of the text q, respectively. The negative sample is the most difficult negative sample extracted from the small batch data .

[0076] Secondly, the formula of the contrastive learning loss is as follows:

[0077]

[0078] Where represents the set of all text negative samples corresponding to the video v in the small batch data , represents the i-th text negative sample, represents the set of all video negative samples corresponding to the text q in the small batch data , represents the i-th video negative sample.

[0079] Finally, the formula of the dynamic distillation loss is as follows:

[0080]

[0081] Where D KL represents the distillation loss function, C s represents the set of similarities between each frame of the video and the text in the student model, and C ta set of similarities between each frame of the video and the text in the representative teacher model.

[0082] As we mentioned, due to the gap in the task-specific domain, the CLIP teacher model can have a huge performance difference on different datasets. At the beginning of training, when the teacher's knowledge is beneficial, learn more knowledge from the teacher, and when the student model becomes stronger, gradually learn more from the existing data. Specifically, in order to obtain more balanced and better distillation results from the dual-branch learning, a dynamic distillation strategy is introduced. The hyperparameter w designed for online adjustment during model training, that is, w = w0g(t), where w0 is the initial weight, t represents the t-th period during training, and g(t) is a decay function.

[0083] Here we use three different types of decay functions to implement the dynamic distillation strategy, which are exponential decay g(t) = k t , linear decay g(t) = kt + b, and Sigmoid decay where k represents the coefficient controlling the decay trend.

[0084] The final loss used when training the inheritance branch is:

[0085]

[0086] where represents the triplet ranking loss of the inheritance branch, represents the contrastive learning loss of the inheritance branch, represents the dynamic distillation loss, and λ1 is a preset weight coefficient. In this embodiment, λ1 = 0.04, w0 = 0.1, and g(t) = 0.95 t to balance the initial weights of each loss at the beginning of training.

[0087] (8-1) Dot-multiply the obtained exploration branch text feature and the feature sequence of the exploration branch video to calculate the cosine similarity between the text feature representation q d and the video feature sequence F d as the similarity between the exploration branch text and each frame of the video, that is, the similarity set C d :

[0088]

[0089] (8-2) Take the maximum value of the similarity between the text and each frame of the video in step (8-1) as the similarity S E (Q, V) between the inheritance branch video and the text, that is:

[0090]

[0091] (9) Through the above steps, the similarity set C of the exploration branch text and the video frame by frame is obtained d and the similarity S of the video and the text E (Q, V) is obtained, and then the exploration branch is trained by using a common space learning algorithm to learn the similarity between the long video and the text. The final loss used when training the exploration branch is :

[0092]

[0093] wherein represents the triplet ranking loss of the exploration branch, represents the contrastive learning loss of the exploration branch, and λ2 is a preset weight coefficient. In the embodiment, λ2 = 0.04 is set to balance the initial weights of each loss at the beginning of training.

[0094] (10-1) The dual-branch dynamic distillation learning network model is trained by steps (7) and (9), and the final loss used when training the dual-branch dynamic distillation learning network is :

[0095]

[0096] wherein represents the loss of the inheritance branch, represents the loss of the exploration branch. The dual-branch dynamic distillation learning network model is trained by .

[0097] (10-2) Through the above steps, a trained dual-branch dynamic distillation learning network model is obtained, and the specific steps of implementing the cross-modal retrieval of text to its related video are as follows: inputting the text and a set of candidate videos, for the text and a candidate video, calculating the similarity S I (Q, V) of the text and the inheritance branch and the similarity S E (Q, V) of the exploration branch and performing weighted sum as the final similarity S(Q, V) of the text and the video, that is:

[0098] S(Q, V) = (1-β)S I (Q, V) + βS E (Q, V)

[0099] wherein β is a preset weight coefficient, and after adjusting the size of β from 0 to 1 in the experiment, the results show that the performance of the network model is best when β is 0.3.

[0100] The text is sorted with the similarity of all candidate videos, and the result with the best similarity is taken as the final returned search result, so as to realize the cross-modal retrieval task of text to its related video.

[0101] Corresponding to the foregoing embodiment of the cross-modal video retrieval method based on double-branch dynamic distillation learning, the present application also provides an embodiment of a cross-modal video retrieval device based on double-branch dynamic distillation learning.

[0102] Referring to Figure 2 The embodiment of the cross-modal video retrieval device based on double-branch dynamic distillation learning provided by the present application comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the cross-modal video retrieval method based on double-branch dynamic distillation learning in the foregoing embodiment.

[0103] The embodiment of the cross-modal video retrieval device based on double-branch dynamic distillation learning of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 2 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device in the embodiment is usually provided with other hardware according to the actual functions of the device with data processing capability where it is located, and the details are not described here. Figure 2 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device in the embodiment is usually provided with other hardware according to the actual functions of the device with data processing capability where it is located, and the details are not described here.

[0104] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and will not be described here.

[0105] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The apparatus embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0106] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the cross-modal video retrieval method based on double-branch dynamic distillation learning in the above embodiment.

[0107] The computer readable storage medium can be an internal storage unit of any data processing device in the above-mentioned embodiments, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0108] The above embodiments are used to explain and illustrate the present application, but not to limit the present application. Any modification and change made to the present application within the spirit and protection scope of the claims falls within the protection scope of the present application.

Claims

1. A cross-modal video retrieval method based on dual-branch dynamic distillation learning, characterized in that, Comprise the following steps: (1) Feature pre-extraction is performed on the query text and the video to be retrieved to obtain text features and video features of the teacher model; (2) Similarity calculation is performed on the text features and video features of the teacher model obtained in step (1) to obtain a semantic similarity distribution as a guide for the inherited branch of the student model; (3) Feature pre-extraction is performed on the query text and the video to be retrieved to obtain initial text features and initial video features of the student model; (4) In the inherited branch of the student model, the initial text features obtained in step (3) are encoded to obtain text feature representations; (5) In the inherited branch of the student model, the initial video features obtained in step (3) are encoded to obtain video feature representations; (6) Similarity calculation is performed on the video feature representations obtained in step (5) and the text feature representations obtained in step (4) to obtain frame-by-frame video and text similarity of the inherited branch, and the maximum value is taken as the video and text similarity of the inherited branch; (7) The video frame-by-frame and text similarity obtained in step (6) and the video and text similarity are used to train the inherited branch through a common space learning algorithm and a dynamic distillation algorithm, so that the inherited branch absorbs beneficial knowledge from the teacher model; (8) The exploration branch and the inherited branch of the student model share the same network architecture, and after the same feature encoding and similarity calculation as steps (4), (5), and (6) of the inherited branch, the video frame-by-frame and text similarity and the video and text similarity of the exploration branch are obtained; (9) The video frame-by-frame and text similarity and the video and text similarity obtained in step (8) are used to train the exploration branch through a common space learning algorithm, so as to learn knowledge on a cross-modal retrieval task specific dataset; (10) The dual-branch dynamic distillation learning network model comprising the inherited branch and the exploration branch is jointly trained through steps (7) and (9), the text and video are input into the trained dual-branch dynamic distillation network model, and cross-modal retrieval of the text to its related video is realized.

2. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, In step (1), different pre-trained models are used to extract text features and video features of the teacher model, including: (1-1) The pre-trained large-scale text feature extractor CLIP model is used to extract the text features of the teacher model; (1-2) The video is divided into video frame sequences, and the pre-trained large-scale picture feature extractor CLIP model is used for feature extraction of the video frame sequences to obtain the video features of the teacher model.

3. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 2, characterized in that, In step (2), the video frame feature sequence obtained in step (1-2) and the text features obtained in step (1-1) are input into the teacher model, and the cosine similarity is calculated for each video frame feature and text feature to obtain a semantic similarity distribution as a guide for the inherited branch of the student model.

4. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, In step (3), different pre-trained models are used to extract text features and video features of the student model, including: (3-1) The pre-trained large text feature extractor RoBERTa model is used to extract a set of word feature vectors to obtain initial text features of the student model; (3-2) dividing the video into a video frame sequence, performing feature extraction on the video frame sequence by using a 2D deep convolutional network ResNet152 pre-trained on ImageNet to obtain initial video features of the student model.

5. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, The method for encoding the initial text features obtained in step (3) includes: the inheritance branch uses a full connection layer and a ReLu activation function to reduce the dimension of the input initial text features, and the reduced text features are input into a transformer after position embedding coding to capture context information, thereby obtaining text feature representation of the inheritance branch; The method for encoding the initial video features obtained in step (3) includes: the inheritance branch uses a full connection layer and a ReLu activation function to reduce the dimension of the input initial video features, and the reduced video features are input into a transformer after position embedding coding to capture context information, and a frame scale feature sequence of the inheritance branch video, i.e., video feature representation of the inheritance branch, is obtained through a full connection layer.

6. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, The method for training the inheritance branch in step (7) specifically includes: The inheritance branch is trained in a dynamic distillation manner; the similarity between the video and the text is learned through a ternary ranking loss and a contrastive learning loss; as the training deepens, the inheritance branch learns more and more knowledge from the teacher model, the weight of the distillation loss is dynamically adjusted, and the beneficial knowledge from the teacher model is absorbed through the dynamic distillation loss.

7. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 6, characterized in that, total loss of training inheritance branch is: wherein, a triplet ranking loss representing the inheritance branch, a contrastive learning loss representing the inheritance branch, a dynamic distillation loss, λ1 is a preset weight coefficient, w is a hyperparameter adjusted online during model training, w=w0g(t), wherein w0 is an initial weight, t represents the tth period during training, g(t) is a decay function, including exponential decay, linear decay, Sigmoid decay; the distillation loss is as follows: wherein k represents the number of video frames, representing the similarity of each frame of the video to the text in the student model inheritance branch, representing the similarity of each frame of the video to the text in the teacher model.

8. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, The method for training the exploration branch in step (9) specifically includes: the exploration branch is trained by using a common space learning algorithm, and the specific properties of the downstream task and the similarity between the video and the text are explored through a ternary ranking loss and a contrastive learning loss.

9. The cross-modal video retrieval method based on dual-branch dynamic distillation learning according to claim 1, characterized in that, In step (10), the loss of the double-branch dynamic distillation learning network is is: wherein, represents a loss of the inheritance branch, represents a loss of the exploration branch.

10. A cross-modal video retrieval device based on dual-branch dynamic distillation learning, comprising a memory and one or more processors, wherein the memory stores executable codes, and the executable codes are characterized in that, The processor, when executing the executable code, implements the cross-modal video retrieval method based on the double-branch dynamic distillation learning according to any one of claims 1-9.

Images