Partial relevant video retrieval method based on context cue mining

Abstract

The application discloses a partial correlation video retrieval method based on context clue mining, comprising the following steps: selecting a partial correlation video retrieval data set and constructing a corresponding query set; constructing a partial correlation video retrieval network; inputting the query text, the video and the query set into a clue mining branch to extract global and local context clues, and calculating the similarity of the text-video pair; constructing a loss function to train the clue mining branch; inputting the query text and the video into a clue distillation branch, simultaneously transferring the mined context clues to the clue distillation branch, and designing an adaptive distillation mechanism to inhibit the diffusion of irrelevant clues; constructing a loss function to train the clue distillation branch; and inputting the query text to be retrieved into the trained clue distillation branch for partial correlation video retrieval. The method solves the problem that the prior art ignores that the query text is only a rough description of the video content, thereby exacerbating the asymmetry between the different modal semantics of the text and the video.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video cross-modal retrieval, and particularly relates to a partial correlation video retrieval method based on context clue mining. BACKGROUND

[0002] With the rapid development of video media platforms such as Douyin, the number of videos is growing exponentially, and users are usually only interested in videos containing specific content, so the demand for accurately retrieving videos with specified content has become increasingly urgent. In most video retrieval scenarios, the query text input by the user is usually only a few keywords that are partially related to the video content, and most videos are long videos that are not cut, so it is difficult to describe the complete content of the video with only a few keywords. Under this background, Dong et al. proposed a partial correlation video retrieval task, which aims to retrieve videos that are only partially related in semantics to the query text. Under this task, a video corresponds to multiple texts, and a text is the corresponding description of a video segment in the video. Due to the partial correlation between the text and the video in semantics, it is more challenging.

[0003] According to different video segment modeling methods, existing partial correlation video retrieval methods can be divided into two categories: explicit video segment modeling and implicit video segment modeling. The former captures video information of different time spans through multiple sliding windows of different scales to explicitly construct the semantic representation of the video segment. Since it depends on the reasonable setting of parameters such as window size and step, this method is prone to introduce a large amount of redundant video segment information, thereby increasing the storage and computing cost. In contrast, the implicit segment modeling method avoids the problem brought by the sliding window. Instead of performing global context semantic extraction on all video frames within a fixed range, this method guides each frame to interact with only its adjacent frames to generate a more multi-scale video feature representation.

[0004] The implicit segment modeling method focuses on the processing of the text, and by decomposing the text query into multiple semantic units, each unit is matched with the video frame in a fine-grained manner to alleviate the problem of semantic granularity mismatch between the text query and the video frame representation. However, this method only performs context semantic interaction within a single text, and the inherent semantic information of a single query text is less than that of the video content, which exacerbates the problem of cross-modal semantic asymmetry, resulting in low accuracy of partial correlation video retrieval. SUMMARY

[0005] The purpose of the present application is to provide a partial correlation video retrieval method based on context clue mining, which solves the problem of ignoring that the query text is only a rough description of the video content, thereby exacerbating the problem of asymmetry between the different modal semantics of the text and the video.

[0006] The technical scheme adopted by the present application is a partial relevant video retrieval method based on context clue mining, which specifically comprises the following steps:

[0007] Step 1: Select a partial relevant video retrieval data set, and construct a corresponding query set for each query text in the data set;

[0008] Step 2: Construct a partial relevant video retrieval network, which includes a clue mining branch and a clue distillation branch;

[0009] Step 3: Input the query text, video and the corresponding query set constructed in step 1 into the clue mining branch to extract global and local context clues, and calculate the text-video pair similarity of the clue mining branch;

[0010] Step 4: Construct a loss function for the clue mining branch to train the clue mining branch;

[0011] Step 5: Input the query text and video in step 3 into the clue distillation branch, and transfer the context clues mined in step 3 to the clue distillation branch, and design an adaptive distillation mechanism to suppress the diffusion of irrelevant clues;

[0012] Step 6: Construct a loss function for the clue distillation branch to train the clue distillation branch;

[0013] Step 7: Input the query text to be retrieved into the trained clue distillation branch for partial relevant video retrieval.

[0014] The present application is also characterized in that:

[0015] Step 1 is specifically:

[0016] Select a data set containing video resources and with corresponding query texts; for any query text containing T words in the data set , construct a query set based on other relevant query texts corresponding to the video V of the query text , wherein is the other relevant query text, and K is the number of other relevant query texts.

[0017] In step 2, the clue mining branch includes a text encoder , a video encoder , a global clue mining module, a local clue mining module and a first similarity measurement module, the text encoder is used to encode the input query text and query set to obtain the encoded query text and encoded query set; the global clue mining module and the local clue mining module calculate global query embedding and local query embedding respectively according to the encoded query text and the encoded query set; the video encoder This module is used to encode the input video to obtain video-level embeddings and segment-level embeddings; the first similarity measurement module is used to calculate the text-video pair similarity.

[0018] Clue distillation branch includes text encoder Video encoder Second similarity measurement module; text encoder Used to encode the input query text to obtain the encoded query text, video encoder. The first module encodes the input video to obtain video-level embeddings and segment-level embeddings; the second similarity measurement module is used to calculate the text-video pair similarity.

[0019] Text encoder for clue mining branch Text encoder for clue distillation branches The structures are the same, and the workflow is as follows:

[0020] For the input query text First, a pre-trained RoBERTa model is used to extract the query text. The features of each word are then mapped to a low-dimensional space using a fully connected neural network activated by ReLU, and learnable positional embeddings are added to obtain the mapped features. Finally, a standard Transformer encoder layer is used to model the context of these mapped features, resulting in... Dimensional context word feature sequence Finally, in An attention module is introduced to aggregate word features and generate the final query text embedding. The specific process is as follows:

[0021] (1)

[0022] In the formula, It is a trainable vector. Represents the attention vector; Represents the normalization function;

[0023] Video encoder for clue mining branch Video encoders for clue distillation branches The structures are the same, and the workflow is as follows:

[0024] For including Uncropped video frames , first, the pre-trained 2D convolutional neural network is used for I3D feature extraction of each video frame to obtain initial video frame features; then, the initial video frame features are modeled into video-level embedding and segment-level embedding through two parallel branches; in the video-level embedding, the initial video frame features are first reduced in dimension through a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with a learnable position embedding is applied to obtain context features , finally, the video embedding is generated using an attention module , and is represented as follows:

[0025] (2)

[0026] wherein, is a trainable vector, represents an attention vector;

[0027] In the segment-level embedding, the initial video frame features are first averaged-pooled; then, a fully connected layer with a ReLU activation function is used to reduce the dimension of the pooled initial video frame features to obtain segment features; finally, the TC-GMMBlock module with a learnable position embedding is applied to the segment features to generate segment-level embedding , wherein is the number of samples, is the feature dimension.

[0028] Step 3 specifically includes the following steps:

[0029] Step 3.1: input the query text and its corresponding query set into the clue mining branch, and extract context information through the text encoder of the clue mining branch, represented as:

[0030] (3)

[0031] wherein, represents the text encoder of the clue mining branch, represents the encoded query text, represents the encoded query set;

[0032] Step 3.2: calculate local query embedding and global query embedding;

[0033] Based on the attention mechanism, a local clue mining module is constructed to calculate the semantic relevance between the current query text and different queries in the query set, and to mine local context clues that supplement the semantics of the current query text. Then, the encoded query text and the local context clues are combined to obtain the local query embedding, represented as:

[0034] (4)

[0035] wherein, , are two learnable linear mapping matrices; denotes a query in the query set;

[0036] The global clue mining module is constructed based on the multi-head self-attention, the encoded query text and the encoded query set are stacked to form a joint representation denoted as , and then the global query embedding is obtained by mining and aggregating the global context clues reflecting the video level through the global clue mining module, and is denoted as follows:

[0037] (5)

[0038] (6)

[0039] wherein, denotes the global query embedding; denotes the average pooling operation, denotes the h th attention head, are the projection matrices of the h th attention head for query, key and value respectively, is the dimension of the key, H is the number of attention heads;

[0040] Step 3.3: The video input clue mining branch is input into the clue mining branch, and the video encoder of the clue mining branch extracts the video-level embedding and the segment-level embedding , and is denoted as follows:

[0041] (7)

[0042] (8)

[0043] Step 3.4: Calculate the global similarity and the local similarity;

[0044] The first similarity pair module calculates the cosine similarity between the global query embedding and the video-level embedding , and defines the global similarity as , which is denoted as follows:

[0045] (9)

[0046] wherein, denotes the global similarity, denotes the cosine similarity;

[0047] The first similarity-to-mass module adopts a max-pooling strategy to compute the local query embedding with the segment-level embedding , defined as the local similarity, denoted as follows:

[0048] (10)

[0049] where denotes the local similarity, denotes the i th video segment, denotes the max-pooling;

[0050] Step 3.5: The text-video pair similarity of the clue mining branch is calculated by weighted summation, denoted as follows:

[0051] (11)

[0052] where denotes the text-video pair similarity of the clue mining branch, denotes the weight balancing the global and local similarities.

[0053] Step 4: The loss function of the clue mining branch is constructed, denoted as follows:

[0054] (12)

[0055] where denotes the baseline loss function, denotes the triplet loss function, denotes the contrastive learning loss function, denotes the query diversification loss function, denotes the best match loss function.

[0056] The triplet loss function is denoted as follows:

[0057] (13)

[0058] where is a boundary constant, , denote the negative query sample with respect to the video V and the negative video sample with respect to the query text , denotes the mini-batch size.

[0059] The contrastive learning loss function is denoted as follows:

[0060] (14)

[0061] wherein, represents all negative query samples in the small batch relative to the video V ; represents all negative video samples in the small batch relative to the query q ;

[0062] The query diversification loss function is represented as follows:

[0063] (15)

[0064] (16)

[0065] wherein, represents a controllable focusing factor, represents a boundary factor, represents a scaling factor, is used to distinguish the difference between multiple query texts in the same video, represents the number of text queries related to a video;

[0066] The best match loss function is represented as follows:

[0067] (17)

[0068] (18)

[0069] wherein, represents the constraint that each query text only matches one video segment ; represents the best match scheme in which all query texts in the same video and video segments have the highest total similarity under the condition that each query text only matches one video segment.

[0070] Step 5 specifically includes the following sub-steps:

[0071] Step 5.1: input the query text in step 3 into the text encoder of the input clue distillation branch to obtain the encoded global query embedding and the local query embedding ; at the same time, input the video in step 3 into the video encoder of the input clue distillation branch to obtain the video-level embedding and the segment-level embedding , which are represented as follows:

[0072] (19) ​

[0073] (20)

[0074] (21)

[0075] (22)

[0076] The second similarity measure module calculates the global similarity, which is represented as follows:

[0077] (23)

[0078] The second similarity measure module calculates the local similarity, which is represented as follows:

[0079] (24)

[0080] Step 5.2: Constructing the text similarity loss by constraining the representation distance between the text embedding of the clue distillation branch and the text embedding of the clue mining branch to transfer the context clues mined in step 3 to the query text; while constructing the visual similarity loss to guide the clue distillation branch to imitate the visual embedding of the clue mining branch.

[0081] The text similarity loss and the visual similarity loss are represented as follows:

[0082] (25)

[0083] (26)

[0084] In the formula, B represents the batch size;

[0085] Step 5.3: Constructing the distribution similarity loss to constrain the consistency of the similarity measure distribution of the two branches, so that the clue distillation branch imitates the clue mining branch at both the feature layer and the alignment layer, and the distribution similarity loss is represented as follows:

[0086] (27)

[0087] In the formula, represents the cross-entropy loss, represents the global similarity matrix of the clue mining and clue distillation branches, represents the local similarity matrix of the clue mining and clue distillation branches.

[0088] Step 5.4: Designing an adaptive distillation mechanism: by adding the cross-entropy loss The learnable module is introduced to generate an adaptive mask matrix m, and the loss term of each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, which is represented as follows:

[0089] (28)

[0090] In the formula, Adaptive mask value at position ;

[0091] To learn the mask matrix m, while learning the model parameters of the clue distillation branch , the adaptive adjustment of the mask is realized by minimizing the difference between the gradient of the reference loss of the clue distillation branch and the gradient of the optimized distribution similarity loss , and the optimization target is set as:

[0092] (29)

[0093] In the formula, , the reference loss function or the optimized distribution similarity loss is represented as:

[0094] The loss function constructed in step 6 is represented as follows:

[0095] (30)

[0096] In the formula, , is a hyperparameter.

[0097] The beneficial effects of the present application are:

[0098] The partial correlation video retrieval method based on context clue mining of the present application supplements the missing video part semantic information of single query by mining the context clues between different queries of the same video, so as to enhance the semantic correlation between query text and video; then, the context clues between different queries are transferred to single query through knowledge distillation, so as to transfer the context clues between different queries to single query semantic; finally, an adaptive distillation mechanism is proposed to dynamically transfer the context clues that have positive enhancement effect on the current query semantic, and the mechanism can inhibit the diffusion of irrelevant clues in the clue distillation process, thereby improving the accuracy of partial correlation video retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0099] Figure 1 is the framework diagram of the partial correlation video retrieval method based on context clue mining of the present application;

[0100] Figure 2 This is a structural diagram of the text encoder in the partially relevant video retrieval method based on contextual clue mining of the present invention;

[0101] Figure 3 This is a structural diagram of the video encoder in the partially related video retrieval method based on contextual clue mining of the present invention;

[0102] Figure 4 This is a framework diagram of the global clue mining module and the local clue mining module in the partially related video retrieval method based on contextual clue mining of the present invention;

[0103] Figure 5 This is a graph showing the decrease in training loss of the clue mining branch in the partially relevant video retrieval method based on contextual clue mining of this invention;

[0104] Figure 6 This is a graph showing the decrease in training loss of the cue distillation branch in the partially relevant video retrieval method based on contextual cue mining in this invention.

[0105] Figure 7 This is a visualization of the retrieval results of the partially related video retrieval method based on contextual clue mining of the present invention and different comparison methods;

[0106] Figure 8 This is a visualization of the retrieval results of the partially related video retrieval method based on contextual clue mining and different comparison methods of the present invention. Detailed Implementation

[0107] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0108] This invention relates to a partially relevant video retrieval method based on contextual clue mining, such as... Figure 1 As shown, the specific steps include the following:

[0109] Step 1: Select a portion of the relevant video retrieval dataset and construct a corresponding query set for each query text within the dataset.

[0110] Specifically, a dataset containing video resources and corresponding query text is selected; for any query text in the dataset containing T words... Construct a query set based on other related query texts of the video V corresponding to the query text. ,in It is the first K is the number of other related query texts.

[0111] Step 2: constructing a partial relevant video retrieval network, the partial relevant video retrieval network comprising a clue mining branch and a clue distillation branch.

[0112] Specifically, as shown in Figure 1 , the clue mining branch comprises a text encoder , a video encoder , a global clue mining module, a local clue mining module, and a first similarity measurement module, the text encoder being configured to encode an input query text and a query set to obtain an encoded query text and an encoded query set, the global clue mining module and the local clue mining module being configured to calculate a global query embedding and a local query embedding respectively according to the encoded query text and the encoded query set, the video encoder being configured to encode an input video to obtain a video-level embedding and a segment-level embedding, and the first similarity measurement module being configured to calculate a text-video pair similarity.

[0113] The clue distillation branch comprises a text encoder , a video encoder , and a second similarity measurement module, the text encoder being configured to encode an input query text to obtain an encoded query text, the video encoder being configured to encode an input video to obtain a video-level embedding and a segment-level embedding, and the second similarity measurement module being configured to calculate a text-video pair similarity.

[0114] In the present application, the text encoder of the clue mining branch and the text encoder of the clue distillation branch have the same structure but independent parameters, and the structure of the text encoder is shown in Figure 2 , and the general workflow is as follows:

[0115] For an input query text , first, a pre-trained RoBERTa model is used to extract the features of each word in the query text ; then, a ReLU-activated fully connected layer neural network is used to map the features of each word to a low-dimensional space, and a learnable position embedding is added to the mapped features to preserve the position information in the text sequence, and then a standard Transformer encoder layer is used to model the context of the mapped features to obtain a -dimensional context word feature sequence ; finally, an attention module is introduced on to aggregate the word features to generate the final query text embedding , and the specific process is represented as follows:

[0116] (1)

[0117] In the formula, It is a trainable vector. Represents the attention vector; This represents the normalization function.

[0118] In this invention, the video encoder for the clue mining branch and the video encoder of the clue distillation branch The structures are the same but the parameters are independent. The specific video encoder structure is as follows: Figure 3 As shown, the general workflow is as follows:

[0119] For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches, thus constructing a multi-granular representation of the video to simultaneously capture local and global semantic information. Segment-level embeddings help the model accurately locate video segments related to the query text, while video-level embeddings are used to measure the global semantic similarity between the complete video content and the query text. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function. Then, the TC-GMMBlock module with learnable positional embeddings from Wang et al.'s 2024 arXiv paper, "GMMFormer v2: An Uncertainty-aware Framework for Partially Relevant Video Retrieval," is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows:

[0120] (2)

[0121] In the formula, It is a trainable vector. Represents the attention vector;

[0122] In the segment-level embedding, multiple initial video frame features are first averaged and pooled. Then, a fully connected layer with ReLU activation is used to reduce the dimensionality of the pooled initial video frame features, yielding segment features. Finally, a TC-GMMBlock module with learnable positional embeddings is applied to the segment features to generate the segment-level embedding. ,in It is the number of samples. is a feature dimension.

[0123] Step 3: input the query text, video and the corresponding query set constructed in step 1 into the clue mining branch to extract global and local context clues and calculate the text-video pair similarity of the clue mining branch. Specifically, the following sub-steps are included:

[0124] Step 3.1: input the query text and its corresponding query set into the clue mining branch, and extract context information through the text encoder of the clue mining branch, which is represented as:

[0125] (3)

[0126] In the formula, denotes the text encoder of the clue mining branch, denotes the encoded query text, denotes the encoded query set.

[0127] Step 3.2: calculate the local query embedding and the global query embedding.

[0128] Different queries reflect information at different times in the video and are associated with each other through context clues to form a complete description of the video content. There are significant differences in the semantic complementarity of each query in the query set to the current query. Therefore, directly taking all queries in the query set as additional input and equally learning their alignment with the video cannot fully play their semantic complementary role. Given the effectiveness of the attention mechanism in modeling different semantic areas, local context clues in different queries are selectively mined according to the degree of semantic complementarity of different queries. Therefore, a local clue mining module is constructed based on the attention mechanism, as shown in Figure 4 , to calculate the semantic correlation between the current query text and different queries in the query set, mine local context clues that complement the semantics of the current query text, and then combine the encoded query text and the local context clues to obtain the local query embedding, which is represented as:

[0129] (4)

[0130] In the formula, , are two learnable linear mapping matrices; denotes a query in the query set;

[0131] Similarly, to measure the global similarity between the query text and the video content, a global clue mining module is constructed based on multi-head self-attention, as shown in Figure 4The context clues reflecting the main line information of the video are dynamically mined and aggregated. Specifically, the encoded query text and the encoded query set are stacked to form a joint representation denoted as , and then global context clues reflecting the video level are mined and aggregated by a global clue mining module to enhance the global semantic similarity between the query text and the video. Among them, the multi-head self-attention mechanism models the semantic association between different queries in different subspaces through multiple independent attention heads; finally, the intermediate results of each attention head are aggregated by an average pooling strategy to obtain a global query embedding reflecting the global semantics of the video, denoted as

[0132] (5)

[0133] (6)

[0134] In the formula, represents the global query embedding; represents the average pooling operation, represents the h th attention head, , and h are the projection matrices of the query, key and value of the th attention head, respectively, H is the dimension of the key, is the number of attention heads;

[0135] Step 3.3: The video input clue mining branch is input into the clue mining branch of the video encoder to extract the video level embedding and the segment level embedding , denoted as

[0136] (7)

[0137] (8)

[0138] Step 3.4: Calculate the global similarity and the local similarity;

[0139] The first similarity pair module calculates the cosine similarity between the global query embedding and the video level embedding , which is defined as the global similarity, denoted as

[0140] (9)

[0141] In the formula, represents the global similarity, represents the cosine similarity;

[0142] The first similarity-to-mass module adopts a max-pooling strategy to compute the local query embedding with segment-level embedding , defined as local similarity, is denoted as follows:

[0143] (10)

[0144] wherein, denotes the local similarity, denotes the i th video segment, denotes the max-pooling;

[0145] Step 3.5: The text-video pair similarity of the clue mining branch is calculated by weighted summation, denoted as follows:

[0146] (11)

[0147] wherein, denotes the text-video pair similarity of the clue mining branch, denotes the weight balancing global and local similarity.

[0148] Step 4: Construct the loss function of the clue mining branch, and train the clue mining branch.

[0149] The clue mining branch aims to improve the semantic similarity between query text and video by mining contextual clues in the query set that can enrich the semantics of the current query. This process is optimized by a benchmark loss function , which is composed of four parts: a triplet loss function , a contrastive learning loss function , a query diversification loss function , and a best match loss function , denoted as follows:

[0150] (12)

[0151] Among them, the triplet loss function is denoted as follows:

[0152] (13)

[0153] wherein, is a boundary constant, , denote the negative query sample relative to the video V and the negative video sample relative to the query text , respectively, denotes the mini-batch size;

[0154] The contrastive learning loss function is represented as follows:

[0155] (14)

[0156] wherein, represents all negative query samples in the small batch with respect to the video V ; represents all negative video samples in the small batch with respect to the query q ;

[0157] The query diversification loss function is represented as follows:

[0158] (15)

[0159] (16)

[0160] wherein, represents a controllable focusing factor, represents a boundary factor, represents a scaling factor, is used to distinguish the difference between multiple query texts in the same video, represents the number of text queries related to one video;

[0161] The best match loss function is represented as follows:

[0162] (17)

[0163] (18)

[0164] wherein, represents a constraint that each query text only matches one video segment ; represents a best match scheme in which all query texts in the same video have the highest total similarity with the video segment under the condition that each query text only matches one video segment.

[0165] Step 5: input the query text and video in step 3 into the clue distillation branch, transfer the context clues mined in step 3 to the clue distillation branch, and design an adaptive distillation mechanism to suppress the diffusion of irrelevant clues.

[0166] The target of the partial-relevant video retrieval task is to input a query text and retrieve the corresponding video of the text. Therefore, the text input of the model during reasoning only has one query text. The clue mining branch can directly mine the context clues between different queries as additional information to enrich the semantics of the current query text. To obtain the context clues between different queries without relying on the query set, the present application transfers the context clues obtained by the clue mining branch to the clue distillation branch through knowledge distillation. Specifically, the following sub-steps are included:

[0167] Step 5.1: input the query text in step 3 into the text encoder of the clue distillation branch Step 5.2: construct the text similarity loss

[0168] (19)

[0169] (20)

[0170] (21)

[0171] (22)

[0172] The second similarity measurement module calculates the global similarity, which is represented as follows:

[0173] (23)

[0174] The second similarity measurement module calculates the local similarity, which is represented as follows:

[0175] (24)

[0176] Step 5.2: construct the text similarity loss by constraining the representation distance between the text embedding of the clue distillation branch and the text embedding of the clue mining branch, so as to transfer the context clues mined in step 3 to the query text; secondly, although the two branches process the same video input, since the video encoding process of the clue mining branch is guided by the query semantics after the fusion of the context clues, while the clue distillation branch only relies on the original query semantics for guidance, there is a significant difference between the learning objectives of the two branches. Therefore, the visual similarity loss ​​​​​​, the guiding clue distillation branch imitates the visual embedding of the clue mining branch, and the offset of the two in the visual representation space is reduced.

[0177] a text similarity loss and a visual similarity loss is represented as follows:

[0178] (25)

[0179] (26)

[0180] wherein, B represents the batch size;

[0181] Step 5.3: To further align the text-video similarity distribution of the clue mining branch and the clue distillation branch in the similarity measure stage of video retrieval, a distribution similarity loss is constructed , which constrains the consistency of the similarity measure distribution of the two branches, so that the clue distillation branch imitates the clue mining branch in the feature layer and the alignment layer at the same time, and the distribution similarity loss is represented as follows:

[0182] (27)

[0183] wherein, represents the cross-entropy loss, represents the global similarity matrix of the clue mining and clue distillation branches, represents the local similarity matrix of the clue mining and clue distillation branches;

[0184] Step 5.4: During the process of mining contextual clues by the clue mining branch, some clues that are irrelevant to the current query may be learned. For example, for the text “A woman is walking along a track” in the query set, although it can supplement the scene clue “along a track” of the query “She takes off at a fast run”, the verb “walking” in it has semantic deviation from the action “run” emphasized in the current query, thereby interfering with the extraction ability of the action semantics in the current query. It can be seen that transferring all contextual clues in the query set to the clue distillation branch will introduce semantic noise and affect the retrieval effect of some related videos. Therefore, the present application uses an adaptive distillation mechanism to suppress the diffusion of irrelevant clues and only transfer some contextual clues that have a positive supplement to the semantics of the current query, so as to adaptively adjust the knowledge distillation process. Specifically, the adaptive distillation mechanism designed by the present application suppresses the diffusion of irrelevant clues by adding a self-adaptive distillation mechanism to the cross-entropy loss The learnable module is introduced to generate an adaptive mask matrix m, and the loss term of each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, which is represented as follows:

[0185] (28)

[0186] In the formula, represents the adaptive mask value at the position .

[0187] To learn the mask matrix m, the model parameters of the clue distillation branch are learned at the same time, and the adaptive adjustment of the mask is realized by minimizing the difference between the gradient of the reference loss of the clue distillation branch and the gradient of the optimized distribution similarity loss , and the optimization target is set as:

[0188] (29)

[0189] In the formula, represents the reference loss function or the optimized distribution similarity loss .

[0190] The adaptive learning mask can effectively suppress the diffusion of irrelevant context clues, and at the same time, it can effectively strengthen the learning of context clues that have a significant effect on query semantic supplement. When the model tends to converge, the learning result of the mask will tend to retain those context clues that are positively related to the query semantics.

[0191] Step 6: Construct the loss function of the clue distillation branch, and train the clue distillation branch.

[0192] The constructed loss function is represented as follows:

[0193] (30)

[0194] In the formula, , is a hyperparameter.

[0195] Step 7: Since the query set cannot be directly obtained, the query text to be retrieved is input into the trained clue distillation branch to perform partial relevant video retrieval.

[0196] Embodiment 1

[0197] The embodiment provides a partial relevant video retrieval method based on context clue mining, which specifically includes the following steps:

[0198] Step 1: select a partial correlation video retrieval dataset, and construct a corresponding query set for each query text in the dataset;

[0199] Step 2: construct a partial correlation video retrieval network, which includes a clue mining branch and a clue distillation branch;

[0200] Step 3: input the query text, video, and the corresponding query set constructed in step 1 into the clue mining branch to extract global and local context clues, and calculate the text-video pair similarity of the clue mining branch;

[0201] Step 4: construct a loss function for the clue mining branch, and train the clue mining branch;

[0202] Step 5: input the query text and video in step 3 into the clue distillation branch, and transfer the context clues mined in step 3 to the clue distillation branch, and design an adaptive distillation mechanism to suppress the diffusion of irrelevant clues;

[0203] Step 6: construct a loss function for the clue distillation branch, and train the clue distillation branch;

[0204] Step 7: input the query text to be retrieved into the trained clue distillation branch for partial correlation video retrieval.

[0205] Embodiment 2

[0206] On the basis of embodiment 1, the public dataset Charades-STA is selected, which contains 6,670 videos and 16,128 text descriptions. The average video duration is about 30 seconds. Each video contains an average of 2.4 time points with corresponding text descriptions. For any query text containing T words in the dataset , a query set is constructed based on other related query texts of the video V corresponding to the query text , where is the other related query text, and K is the number of other related query texts.

[0207] Embodiment 3

[0208] On the basis of embodiment 2, in step 2, the clue mining branch includes a text encoder , a video encoder , a global clue mining module, a local clue mining module, and a first similarity measurement module. The text encoder is used to encode the input query text and query set to obtain the encoded query text and encoded query set; the global clue mining module and the local clue mining module calculate global query embedding and local query embedding, respectively, according to the encoded query text and the encoded query set; the video encoder The first similarity measure module is configured to calculate the text-video pair similarity.

[0209] The clue distillation branch includes a text encoder , a video encoder and a second similarity measure module; the text encoder is configured to encode the input query text to obtain an encoded query text, the video encoder is configured to encode the input video to obtain video-level embeddings and segment-level embeddings; and the second similarity measure module is configured to calculate the text-video pair similarity.

[0210] The text encoder of the clue mining branch and the text encoder of the clue distillation branch have the same structure, and the workflow is as follows:

[0211] For the input query text , first, a pre-trained RoBERTa model is used to extract the features of each word in the query text , and the extracted features are 1024-dimensional; then, a fully connected layer neural network with ReLU activation is used to map the features of each word to a low-dimensional space, and a learnable position embedding is added to obtain the mapped features, and then a standard Transformer encoder layer is used to model the context of the mapped features to obtain a sequence of context word features ; finally, an attention module is introduced on to aggregate the word features to generate the final query text embedding , and the specific process is represented as follows:

[0212] (1)

[0213] In the formula, is a trainable vector, represents an attention vector; represents a normalization function;

[0214] The video encoder of the clue mining branch and the video encoder of the clue distillation branch have the same structure, and the workflow is as follows:

[0215] For an uncropped video containing video frames ​, first, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features; then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches; in the video-level embedding, first, the initial video frame features are reduced in dimension through a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with a learnable position embedding is applied to obtain context features , finally, a video embedding is generated using an attention module , which is represented as follows:

[0216] (2)

[0217] wherein, is a trainable vector, denotes an attention vector;

[0218] In the segment-level embedding, first, the initial video frame features are averaged-pooled; then, a fully connected layer with a ReLU activation function is used to reduce the dimension of the pooled initial video frame features to obtain segment features; finally, a TC-GMMBlock module with a learnable position embedding is applied to the segment features to generate segment-level embeddings , wherein is the number of samples, is the feature dimension.

[0219] Embodiment 4

[0220] Based on embodiment 3, step 3 specifically includes the following steps:

[0221] Step 3.1: input the query text and its corresponding query set into the clue mining branch, and extract context information through the text encoder of the clue mining branch, which is represented as:

[0222] (3)

[0223] wherein, denotes the text encoder of the clue mining branch, denotes the encoded query text, denotes the encoded query set;

[0224] Step 3.2: calculate local query embeddings and global query embeddings;

[0225] A local clue mining module is constructed based on an attention mechanism. It calculates the semantic relevance between the current query text and different queries in the query set, and mines local contextual clues that supplement the semantics of the current query text. Then, the encoded query text and local contextual clues are combined to obtain the local query embedding, as shown below:

[0226] (4)

[0227] In the formula, , These are two learnable linear mapping matrices; Refers to a query within a query set;

[0228] A global clue mining module is constructed based on multi-head self-attention, which stacks the encoded query text and the encoded query set to form a joint representation denoted as . Then, the global clue mining module mines and aggregates global contextual clues reflecting the video level to obtain the global query embedding, as shown below:

[0229] (5)

[0230] (6)

[0231] In the formula, Indicates global query embedding; This indicates the average pooling operation. Indicates the first h One's attention, The first h The projection matrix of the query, key, and value of each attention head. It is the dimension of the key. H It's about the number of heads;

[0232] Step 3.3: Input the video into the clue mining branch, and pass it through the video encoder of the clue mining branch. Extracting video-level embeddings and fragment-level embedding , means as follows:

[0233] (7)

[0234] (8)

[0235] Step 3.4: Calculate global similarity and local similarity;

[0236] The first similarity logarithmic module calculates the global query embedding. and video-level embedding The cosine similarity, defined as the global similarity, is expressed as follows:

[0237] (9)

[0238] wherein, denotes global similarity, denotes cosine similarity;

[0239] The first similarity pair module calculates the similarity of local query embedding using the max-pooling strategy and segment-level embedding , defined as local similarity, denoted as follows:

[0240] (10)

[0241] wherein, denotes local similarity, denotes the i th video segment, denotes max-pooling;

[0242] Step 3.5: Calculate the text-video pair similarity of the clue mining branch by weighted summation, denoted as follows:

[0243] (11)

[0244] wherein, denotes the text-video pair similarity of the clue mining branch, denotes the weight of balancing global and local similarity, and the embodiment ;

[0245] Step 4: Construct the loss function of the clue mining branch, denoted as follows:

[0246] (12)

[0247] wherein, denotes the baseline loss function, denotes the triplet loss function, denotes the contrastive learning loss function, denotes the query diversification loss function, denotes the best match loss function;

[0248] The triplet loss function is denoted as follows:

[0249] (13)

[0250] wherein, is a boundary constant, , respectively denote the video VNegative query samples and relative to query text Negative video samples, Indicates the size of the small batch;

[0251] The contrastive learning loss function is expressed as follows:

[0252] (14)

[0253] In the formula, Indicates a small batch relative to the video V All negative query samples, Indicates the relationship between a small batch and a query. q All negative video samples;

[0254] The query for diversified loss functions is as follows:

[0255] (15)

[0256] (16)

[0257] In the formula, This indicates a controllable focusing factor. Represents the boundary factor. Indicates the scaling factor. Used to distinguish the differences between multiple query texts in the same video. This indicates the number of text queries related to a video.

[0258] The optimal matching loss function is expressed as follows:

[0259] (17)

[0260] (18)

[0261] In the formula, This indicates that each query text is constrained. Match only one video clip ; This represents the best matching scheme that maximizes the total similarity between all query texts and video segments within the same video, assuming each query text matches only one video segment.

[0262] Example 5

[0263] Based on Example 4, step 5 specifically includes the following sub-steps:

[0264] Step 5.1: Extract the query text from Step 3. Text encoder for input clue distillation branch Obtain the encoded global query embedding and local query embedding ; meanwhile, the video encoder of the video input clue distillation branch in step 3 to get video-level embedding and segment-level embedding , which is expressed as follows:

[0265] (19)

[0266] (20)

[0267] (21)

[0268] (22)

[0269] The second similarity measure module calculates the global similarity, which is expressed as follows:

[0270] (23)

[0271] The second similarity measure module calculates the local similarity, which is expressed as follows:

[0272] (24)

[0273] Step 5.2: Constructing text similarity loss by constraining the representation distance between the text embedding of the clue distillation branch and the text embedding of the clue mining branch to transfer the context clues mined in step 3 to the query text; meanwhile, constructing visual similarity loss to guide the clue distillation branch to imitate the visual embedding of the clue mining branch.

[0274] Text similarity loss and visual similarity loss are expressed as follows:

[0275] (25)

[0276] (26)

[0277] In the formula, B represents the batch size;

[0278] Step 5.3: Constructing distribution similarity loss to constrain the consistency of the similarity measure distribution of the two branches, so that the clue distillation branch imitates the clue mining branch at both the feature layer and the alignment layer, and the distribution similarity loss is expressed as follows:

[0279] (27)

[0280] wherein, represents the cross-entropy loss, represents the global similarity matrix of the clue mining and clue distillation branches, represents the local similarity matrix of the clue mining and clue distillation branches;

[0281] Step 5.4: Designing an adaptive distillation mechanism: by introducing a learnable module in the cross-entropy loss , an adaptive mask matrix m is generated, and the loss term of each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, which is represented as follows:

[0282] (28)

[0283] wherein, represents the adaptive mask value at position ;

[0284] To learn the mask matrix m, while learning the model parameters of the clue distillation branch, the adaptive adjustment of the mask is realized by minimizing the difference between the gradient of the reference loss of the clue distillation branch and the gradient of the optimized distribution similarity loss , and the optimization objective is set as:

[0285] (29)

[0286] wherein, represents the reference loss function or the optimized distribution similarity loss of the gradient value with respect to the model parameters of the clue distillation branch.

[0287] Embodiment 6

[0288] On the basis of embodiment 5, the loss function constructed in step 6 is represented as follows:

[0289] (30)

[0290] wherein, , is a hyperparameter, = 0.2, = 1.0;

[0291] Simulation experiment

[0292] The simulation experiment is carried out on a single RTX 4090 GPU on the basis of embodiment 6, the training and testing of the model are carried out by using Python 3.8 programming language and PyTorch 2.0.0 framework, and the calculation acceleration is supported by CUDA 11.8 and cuDNN8.0. In terms of experimental parameters, the model learning rate lr is set to 0.00025.

[0293] The evaluation index R@K based on ranking is used, K is in {1, 5, 10, 100}, and is shown in formula (31), which measures the proportion of successfully retrieving videos corresponding to the query in the first K retrieval results, expressed in percentage, the larger the R@K value, the better the method performance. K K In addition, the sum of R@K under different values is measured by SumR to evaluate the retrieval performance as a whole.

[0294] (31)

[0295] In order to verify the effectiveness of the method of the application, the (short) video retrieval method RIVRL, the video corpus moment retrieval method XML, ReLoCLNet and CONQUER, and the partial correlation video retrieval method MS-SL, PEAN, T-D3N, GMM and GMM-V2 are compared.

[0296] Experimental results:

[0297] The retrieval performance of the partial correlation video retrieval method of the application and different comparison methods is shown in Table 1. Figure 3 As can be seen from Table 1, the (short) video retrieval method performs the worst, because this method measures the complete correlation between text and video content, which limits its ability to retrieve partial correlation videos. Taking the RIVRL method as an example, the SUMR is only 54.3%, which is the lowest among all methods. The purpose of the video corpus moment retrieval method is to retrieve the moment information related to the query text, which helps to learn the partial correlation between text and video to some extent. Compared with the mainstream partial correlation video retrieval method, the method of the application achieves the most advanced performance, and the SUMR index is improved by 2.3% compared with the optimal partial correlation video retrieval method.

[0298] Table 1

[0299]

[0300] ​From the above experimental analysis, the global context clues mined in the method of the application are helpful to find the potential information associated with the front and back in the video content to measure the global text-video similarity. The local context clues can supplement the missing video details in the single query text, and then promote the model to perform local alignment at the segment level. This shows the effectiveness and innovation of the method of the application in the aspect of context clue mining for partial relevant video retrieval. At the same time, through the adaptive distillation mechanism, the context clues that have a positive enhancement effect on the current query semantics are transferred, instead of all the clues. This effectively avoids the introduction of redundant information in the distillation process, and solves the problem that all query texts of a video cannot be obtained in the actual retrieval process, thereby improving the robustness of retrieval.

[0301] Table 2 and Table 3 respectively show the retrieval performance diagrams of different global and local context clue mining methods in the method of the application. Feature concatenation fuses text semantics by concatenating multiple query text features in the same dimension. Max-pooling helps the model to fuse key semantic information from different queries by extracting the maximum value in the local region of the query text feature. The attention method dynamically mines the semantic association between queries by assigning a weight coefficient between 0 and 1 to each word of each query text.

[0302] Table 2

[0303]

[0304] Table 3

[0305]

[0306] As can be seen from Table 2 and Table 3, the attention method can selectively aggregate text context clues that have a positive complementary effect on the query text semantics and have a finer granularity, thereby better alleviating the semantic asymmetry problem between the two modalities and achieving the best performance. Taking local context clue mining as an example, the SUMR index of the attention method is 31.7% and 16.5% higher than that of the feature concatenation and max-pooling methods, respectively. In contrast, the simple concatenation of different query features treats the degree of complement of each query to the current query semantics equally, but not every word in each query has a positive semantic complement to the current query, so its effect is the worst. The max-pooling method can highlight some context clues with strong semantic relevance, but it cannot dynamically focus on the importance of different positions, and may miss some key context clue information. In addition, both methods may introduce other irrelevant or redundant information, affecting the overall performance of the model.

[0307] Figure 5 and Figure 6The training loss of the clue mining branch and the clue distillation branch in the method of the application is visualized respectively. As can be seen from the figure, the clue mining branch tends to be stable after 800 iterations, and the loss value of the clue distillation branch tends to be stable after 2000 iterations. The experimental results show that the model in the method of the application can converge effectively, and there is no overfitting or local optimal solution.

[0308] Figure 7 and Figure 8 The retrieval result visualization comparison of the method of the application and other representative part-related video retrieval methods MS-SL and GMM-V2 is shown. The video marked with a frame in the figure is the real video corresponding to the query text. Figure 7 In the figure, although the MS-SL method successfully captures the action semantics "takes" in the query text, it ignores the key semantics "blanket", resulting in its top-3 retrieval results being positioned to similar but irrelevant videos. The GMM-V2 method alleviates this problem to some extent through good video segment modeling capability, but still has retrieval deviation. In contrast, the method of the application can effectively distinguish these wrong matching videos by mining the context clues such as "door", "closet" and "light" implied in the query set. The query text semantics after fusing the above context clues can more accurately describe the video content, so as to accurately retrieve the target video. Figure 8 Similar phenomenon also occurs, further verifying the effectiveness of the context clue mining method proposed in the application in improving the semantic similarity between the query text and the video.

Claims

1. A partial-relevant video retrieval method based on context cue mining, characterized in that, Specifically comprising the following steps: Step 1: Selecting a partial correlation video retrieval data set, constructing a corresponding query set for each query text in the data set; Step 2: Constructing a partial correlation video retrieval network, the partial correlation video retrieval network comprising a clue mining branch and a clue distillation branch; The clue mining branch includes a text encoder , a video encoder , a global clue mining module, a local clue mining module, and a first similarity measurement module, the text encoder is configured to encode the input query text and the query set to obtain encoded query text and encoded query set; the global clue mining module and the local clue mining module are configured to calculate global query embedding and local query embedding according to the encoded query text and the encoded query set respectively; the video encoder is configured to encode the input video to obtain video-level embedding and segment-level embedding; and the first similarity measurement module is configured to calculate text-video pair similarity. The clue distillation branch comprises a text encoder , a video encoder and a second similarity measure module; the text encoder is configured to encode input query text to obtain encoded query text, the video encoder is configured to encode input video to obtain video-level embedding and segment-level embedding; and the second similarity measure module is configured to calculate text-video pair similarity. Step 3: Inputting the query text, video and corresponding query set constructed in step 1 into the clue mining branch to extract global and local context clues, and calculating the text-video pair similarity of the clue mining branch; Step 4: Constructing a loss function of the clue mining branch to train the clue mining branch; Step 5: Inputting the query text and video in step 3 into the clue distillation branch, transferring the context clues mined in step 3 to the clue distillation branch, and designing an adaptive distillation mechanism to suppress the diffusion of irrelevant clues; Step 6: Constructing a loss function of the clue distillation branch to train the clue distillation branch; Step 7: Inputting the query text to be retrieved into the trained clue distillation branch for partial correlation video retrieval.

2. The method for partial relevant video retrieval based on context cues mining according to claim 1, wherein, Step 1 is specifically: A dataset comprising video resources and with corresponding query texts is selected; for any one query text comprising T words in the dataset , a query set is constructed based on other related query texts of the video V corresponding to the query text , wherein is the th other related query text, and K is the number of other related query texts.

3. The partial correlation video retrieval method based on context clue mining according to claim 1, wherein, The text encoder of the lead mining branch The text encoder of the lead distillation branch The structure is the same, and the workflow is as follows: For the input query text , first, the pre-trained RoBERTa model is used to extract the feature of each word in the query text ; then, the feature of each word is mapped to a low-dimensional space by a fully connected layer neural network with ReLU activation and added with a learnable position embedding to obtain the mapped feature, and then a standard Transformer encoder layer is used to model the context of the mapped feature to obtain a sequence of context word features in -dimensional space; finally, an attention module is introduced on to aggregate the word features to generate the final query text embedding , the specific process is as follows: （1） wherein is a trainable vector, denotes an attention vector; denotes a normalization function; Video encoder of the clue mining branch Video encoder of the clue distillation branch The structure is the same, and the workflow is as follows: For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with learnable positional embeddings is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows: （2） wherein is a trainable vector, denotes an attention vector; In the segment-level embedding, firstly, average pooling is performed on the initial video frame features; then, a fully connected layer with ReLU activation function is used to reduce the dimension of the pooled initial video frame features to obtain segment features; finally, the TC-GMMBlock module with learnable position embedding is applied to the segment features to generate the segment-level embedding wherein is the number of samples, is the feature dimension.

4. The method of claim 3, wherein, Step 3 specifically comprises the following steps: Step 3.1: Query text and its corresponding query set Input the clue mining branch, and the context information is extracted by the text encoder of the clue mining branch, which is represented as: （3） In the formula, a text encoder representing a clue mining branch, an encoded query text, an encoded query set; Step 3.2: Calculate the local query embedding and the global query embedding; A local clue mining module is constructed based on an attention mechanism to calculate the semantic correlation between the current query text and different queries in the query set, mine local context clues that supplement the semantics of the current query text, and then combine the encoded query text and the local context clues to obtain a local query embedding, which is represented as follows: （4） wherein , are two learnable linear mapping matrices; denotes one query in the query set; A global clue mining module is constructed based on multi-head self-attention, and the encoded query text and the encoded query set are stacked to form a joint representation denoted as , and then the global clue mining module is used to mine and aggregate the global context clues reflecting the video level to obtain a global query embedding, which is denoted as follows: （5） （6） wherein, denotes a global query embedding; denotes an average pooling operation, denotes the h th attention head, are the projection matrices for the query, key and value of the h th attention head, respectively, is the dimension of the key, H is the number of attention heads; Step 3.3: Video input cue mining branch, by cue mining branch video encoder extracting video-level embeddings and segment-level embeddings , is represented as follows: （7） （8） Step 3.4: Calculate the global similarity and the local similarity; The first similarity measure module computes a global query embedding and a video-level embedding The cosine similarity of the global query embedding and the video-level embedding, defined as a global similarity, is denoted as follows: （9） In the formula, represents the global similarity, represents the cosine similarity; The first similarity measure module employs a max-pooling strategy to compute local query embeddings with segment-level embeddings The similarity with the segment-level embeddings, defined as local similarity, is denoted as follows: （10） wherein denotes a local similarity, denotes the i video segment, denotes max pooling; Step 3.5: Calculate the text-video pair similarity of the clue mining branch by weighted summation, which is represented as follows: （11） wherein, represents the text-video pair similarity of the clue mining branch, represents the weight balancing global and local similarities.

5. The method for partial relevant video retrieval based on context cues mining according to claim 1, wherein, The loss function of the clue mining branch constructed in step 4 is represented as follows: （12） wherein denotes the baseline loss function, denotes the triple loss function, denotes the contrastive learning loss function, denotes the query diversification loss function, denotes the best match loss function; The triplet loss function is represented as follows: （13） wherein, is a boundary constant, , respectively represent negative query samples with respect to videos V and negative video samples with respect to query texts , denotes a mini-batch size; The contrastive learning loss function is represented as follows: （14） wherein represents all negative query samples in the mini-batch with respect to the video V represents all negative query samples in the mini-batch with respect to the video represents all negative video samples in the mini-batch with respect to the query q represents all negative video samples in the mini-batch with respect to the query The query diversification loss function is represented as follows: （15） （16） wherein, denotes a controllable focusing factor, denotes a boundary factor, denotes a scaling factor, for distinguishing the difference between multiple query texts in the same video, denotes the number of text queries related to a video; The best match loss function is represented as follows: （17） （18） wherein represents to constrain each query text to match only one video segment ; represents to let the best matching scheme of all query texts and video segments in the same video with the highest total similarity of query texts and video segments in the case of each query text matching only one video segment.

6. The method for partial relevant video retrieval based on context cue mining according to claim 5, wherein, Step 5 specifically comprises the following sub-steps: Step 5.1: encode the query text from step 3 into a text embedding Text encoder of the input cue distillation branch to obtain a global query embedding and a local query embedding Step 5.2: encode the video input cue distillation branch from step 3 into a video embedding to obtain a video-level embedding and a segment-level embedding , denoted as follows: （19） （20） （21） （22） The second similarity measurement module calculates the global similarity, which is represented as follows: （23） The second similarity measurement module calculates the local similarity, which is represented as follows: （24） Step 5.2: Constructing the text similarity loss by constraining the representation distance between the text embeddings of the cue distillation branch and the cue mining branch text embeddings to transfer the mined contextual cues from step 3 to the query text; At the same time, a visual similarity loss is constructed The guiding clue distillation branch imitates the visual embedding of the clue mining branch. the text similarity loss and the visual similarity loss is represented as follows: （25） （26） In the formula, B represents the batch size; Step 5.3: Constructing Distribution Similarity Loss , constraining the consistency of the two branch similarity measure distributions, so that the clue distillation branch simultaneously imitates the clue mining branch at both the feature layer and the alignment layer, the distribution similarity loss is represented as follows: （27） wherein denotes the cross-entropy loss, denotes the global similarity matrix of the clue mining and clue distillation branches, denotes the local similarity matrix of the clue mining and clue distillation branches; Step 5.4: Designing an adaptive distillation mechanism: By weighting the loss terms for each position of the cross-entropy loss Lce with a learnable mask matrix m, the distribution similarity loss Ldis is generated as follows: The learnable mask matrix m is introduced as follows: （28） In the formula, indicates the adaptive mask value at the position ​ For learning the mask matrix m, the model parameters of the learning clue distillation branch are learned at the same time, by minimizing the difference between the gradient of the reference loss of the clue distillation branch and the gradient of the optimized distribution similarity loss of the mask, the adaptive adjustment of the mask is realized, and the optimization target is represented as: （29） In the formula, denotes the reference loss function or the optimized distribution similarity loss The gradient value of the clue distillation branch model parameter .

7. The method according to claim 6, wherein, The loss function constructed in step 6 is represented as follows: （30） wherein , are hyperparameters.

Images