Partial correlation video retrieval method based on context clue mining

Abstract

The invention discloses a partially related video retrieval method based on context clue mining. The method comprises the following steps: selecting a partially related video retrieval data set and constructing a corresponding query set; constructing a partially related video retrieval network; inputting a query text, a video and a query set into a clue mining branch to extract global and local context clues, and calculating the similarity of a text-video pair; constructing a loss function training clue mining branch; inputting a query text and a video into a clue distillation branch, transferring mined context clues to the clue distillation branch, and designing a self-adaptive distillation mechanism to inhibit diffusion of irrelevant clues; constructing a loss function to train clue distillation branches; and inputting a query text to be retrieved into the trained clue distillation branch to perform partial related video retrieval. According to the method, the problem that asymmetry between different modal semantics of the text and the video is aggravated due to the fact that query texts are ignored and only video contents are roughly described in the prior art is solved.

Description

Technical Field

[0001] This invention belongs to the field of video cross-modal retrieval technology, specifically relating to a method for retrieving partially relevant videos based on contextual clue mining. Background Technology

[0002] With the rapid development of video media platforms such as Douyin (TikTok), the number of videos has grown exponentially. Users are typically only interested in videos containing specific content, making the need for accurate retrieval of videos with specific content increasingly urgent. In most video retrieval scenarios, the query text entered by users is usually just a few keywords partially related to the video content. However, most videos are long, unedited videos, making it difficult to describe the complete content of a video using only a few keywords. Against this backdrop, Dong et al. proposed a partially related video retrieval task, aiming to retrieve videos that are only partially semantically related to the query text. In this task, one video corresponds to multiple texts, and each text is a description of a specific video segment. Due to the partial semantic relevance between the text and the video, it presents a greater challenge.

[0003] Based on different video segment modeling methods, existing related video retrieval methods can be divided into two categories: explicit video segment modeling and implicit video segment modeling. The former captures video information across different time spans using multiple sliding windows of varying scales to explicitly construct semantic representations of video segments. Because this relies on the proper setting of parameters such as window size and stride, it easily introduces a large amount of redundant video segment information, thus increasing storage and computational costs. In contrast, methods based on implicit segment modeling avoid the problems associated with sliding windows. These methods do not perform global contextual semantic extraction on all video frames within a fixed range; instead, they guide each frame to dynamically interact with only its neighboring frames to generate more multi-scale video feature representations.

[0004] Some methods based on implicit fragment modeling focus on text processing. They decompose text queries into multiple semantic units, each of which is then finely matched with video frames to alleviate the semantic granularity mismatch between text queries and video frame representations. However, this method only performs contextual semantic interaction within the query text itself. The inherent semantic information of a single query text is less than that of the video content, which exacerbates the problem of cross-modal semantic asymmetry, resulting in low accuracy in retrieving some relevant videos. Summary of the Invention

[0005] The purpose of this invention is to provide a partially relevant video retrieval method based on contextual clue mining, which solves the problem in the prior art that ignores the query text as only a rough description of the video content, thereby exacerbating the asymmetry between the semantics of different modalities of text and video.

[0006] The technical solution adopted in this invention is a partially relevant video retrieval method based on contextual clue mining, which specifically includes the following steps: Step 1: Select a portion of the relevant video retrieval dataset and construct a corresponding query set for each query text within the dataset; Step 2: Construct a partially relevant video retrieval network, which includes a clue mining branch and a clue distillation branch; Step 3: Input the query text, video, and the corresponding query set constructed in Step 1 into the clue mining branch to extract global and local context clues, and calculate the text-video pair similarity of the clue mining branch; Step 4: Construct the loss function for the clue mining branch and train the clue mining branch; Step 5: Distill the query text and video input clues from Step 3 into the clue distillation branch, and transfer the context clues mined in Step 3 to the clue distillation branch. Design an adaptive distillation mechanism to suppress the spread of irrelevant clues. Step 6: Construct the loss function for the cue distillation branch and train the cue distillation branch; Step 7: Input the query text to be retrieved into the trained cue distillation branch to perform partial related video retrieval.

[0007] The invention is further characterized by: Step 1 is as follows: Select a dataset containing video resources and corresponding query text; for any query text in the dataset containing T words... Construct a query set based on other related query texts of the video V corresponding to the query text. ,in It is the first K is the number of other related query texts.

[0008] In step 2, the clue mining branch includes a text encoder. Video encoder The system includes a global clue mining module, a local clue mining module, a first similarity measurement module, and a text encoder. This module encodes the input query text and query set to obtain encoded query text and encoded query set; the global clue mining module and the local clue mining module calculate the global query embedding and local query embedding based on the encoded query text and encoded query set, respectively; the video encoder... This module is used to encode the input video to obtain video-level embeddings and segment-level embeddings; the first similarity measurement module is used to calculate the text-video pair similarity. Clue distillation branch includes text encoder Video encoder Second similarity measurement module; text encoder Used to encode the input query text to obtain the encoded query text, video encoder. The first module encodes the input video to obtain video-level embeddings and segment-level embeddings; the second similarity measurement module is used to calculate the text-video pair similarity.

[0009] Text encoder for clue mining branch Text encoder for clue distillation branches The structures are the same, and the workflow is as follows: For the input query text First, a pre-trained RoBERTa model is used to extract the query text. The features of each word are then mapped to a low-dimensional space using a fully connected neural network activated by ReLU, and learnable positional embeddings are added to obtain the mapped features. Finally, a standard Transformer encoder layer is used to model the context of these mapped features, resulting in... Dimensional context word feature sequence Finally, in An attention module is introduced to aggregate word features and generate the final query text embedding. The specific process is as follows: (1) In the formula, It is a trainable vector. Represents the attention vector; Represents the normalization function; Video encoder for clue mining branch and the video encoder of the clue distillation branch The structures are the same, and the workflow is as follows: For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with learnable positional embeddings is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows: (2) In the formula, It is a trainable vector. Represents the attention vector; In the segment-level embedding, the initial video frame features are first subjected to average pooling; then, a fully connected layer with ReLU activation is used to reduce the dimensionality of the pooled initial video frame features to obtain segment features; finally, the TC-GMMBlock module with learnable position embedding is applied to the segment features to generate the segment-level embedding. ,in It is the number of samples. It is the feature dimension.

[0010] Step 3 specifically includes the following steps: Step 3.1: Submit the query text and its corresponding query set The input clue mining branch is used to extract contextual information through the text encoder of the clue mining branch, which is represented as follows: (3) In the formula, The text encoder representing the clue mining branch, Represents the encoded query text, This represents the encoded query set; Step 3.2: Calculate the local query embedding and the global query embedding; A local clue mining module is constructed based on an attention mechanism. It calculates the semantic relevance between the current query text and different queries in the query set, and mines local contextual clues that supplement the semantics of the current query text. Then, the encoded query text and local contextual clues are combined to obtain the local query embedding, as shown below: (4) In the formula, , These are two learnable linear mapping matrices; Refers to a query within a query set; A global clue mining module is constructed based on multi-head self-attention, which stacks the encoded query text and the encoded query set to form a joint representation denoted as . Then, the global clue mining module mines and aggregates global contextual clues reflecting the video level to obtain the global query embedding, as shown below: (5) (6) In the formula, Indicates global query embedding; This indicates the average pooling operation. Indicates the first h One point of attention, The first h The projection matrix of the query, key, and value of each attention head. It is the dimension of the key. H It's about the number of heads; Step 3.3: Input the video into the clue mining branch, and pass it through the video encoder of the clue mining branch. Extracting video-level embeddings and fragment-level embedding , means as follows: (7) (8) Step 3.4: Calculate global similarity and local similarity; The first similarity logarithmic module calculates the global query embedding. and video-level embedding The cosine similarity, defined as the global similarity, is expressed as follows: (9) In the formula, Indicates global similarity. This indicates the calculation of cosine similarity. The first similarity logarithmic module uses a max-pooling strategy to calculate the local query embedding. With fragment-level embedding The similarity between them is defined as local similarity, and is expressed as follows: (10) In the formula, Indicates local similarity. Indicates the first i A video clip, Indicates max pooling; Step 3.5: Calculate the text-video pair similarity of the clue mining branch using a weighted summation method, as shown below: (11) In the formula, The text-video pair similarity represents the branching of clue mining. This represents the weight that balances global and local similarity.

[0011] The loss function for the clue mining branch constructed in step 4 is expressed as follows: (12) In the formula, Represents the baseline loss function. Represents the triplet loss function. This represents the contrastive learning loss function. This indicates a query for a diversified loss function. This represents the optimal matching loss function; The triplet loss function is expressed as follows: (13) In the formula, It is a boundary constant. , These respectively represent relative to video. V Negative query samples and relative to query text Negative video samples, Indicates the size of the small batch; The contrastive learning loss function is expressed as follows: (14) In the formula, Indicates a small batch relative to the video V All negative query samples, Indicates the difference between a small batch and a query. q All negative video samples; The query for diversified loss functions is as follows: (15) (16) In the formula, This indicates a controllable focusing factor. Represents the boundary factor. Indicates the scaling factor. Used to distinguish the differences between multiple query texts in the same video. This indicates the number of text queries related to a video. The optimal matching loss function is expressed as follows: (17) (18) In the formula, This indicates that each query text is constrained. Match only one video clip ; This represents the best matching scheme that maximizes the total similarity between all query texts and video segments within the same video, assuming each query text matches only one video segment.

[0012] Step 5 specifically includes the following sub-steps: Step 5.1: Extract the query text from Step 3. Text encoder for input clue distillation branch Obtain the encoded global query embedding and local query embedding Simultaneously, the video from step 3 is input into the video encoder of the clue distillation branch. Get video-level embedding and fragment-level embedding , means as follows: (19) (20) (twenty one) (twenty two) The second similarity measurement module calculates the global similarity, as shown below: (twenty three) The second similarity measurement module calculates local similarity, as shown below: (twenty four) Step 5.2: Constructing the text similarity loss. By constraining the representation distance between the text embeddings of the cue distillation branch and the text embeddings of the cue mining branch, the contextual cues mined in step 3 are transferred to the query text; simultaneously, a visual similarity loss is constructed. The visual embedding of clue distillation branches mimics the clue mining branches; Text similarity loss and visual similarity loss It is expressed as follows: (25) (26) In the formula, B Represents batch size; Step 5.3: Construct the distribution similarity loss To ensure consistency in the distribution of similarity metrics between the two branches, the clue distillation branch simultaneously mimics the clue mining branch in both the feature layer and the alignment layer, thus distributing the similarity loss. It is expressed as follows: (27) In the formula, Represents cross-entropy loss, This represents the global similarity matrix between the clue mining and clue distillation branches. Represents the local similarity matrix between the clue mining and clue distillation branches; Step 5.4: Design an adaptive distillation mechanism: through cross-entropy loss A learnable module is introduced to generate an adaptive mask matrix m. The loss term at each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, as shown below: (28) In the formula, Indicates position The adaptive mask value at the location; To learn the mask matrix m, the model parameters of the learning cue distillation branch are... At the same time, by minimizing the baseline loss of the clue distillation branch. gradient and optimized distribution similarity loss The difference between gradients is used to achieve adaptive adjustment of the mask. The optimization objective is expressed as: (29) In the formula, This indicates the calculation of the baseline loss function. Or optimized distribution similarity loss Regarding the gradient value of parameter 𝜃 in the clue distillation branch model.

[0013] The loss function constructed in step 6 is expressed as follows: (30) In the formula, , It's a hyperparameter.

[0014] The beneficial effects of this invention are: This invention presents a partially relevant video retrieval method based on contextual clue mining. It supplements the missing semantic information of a single query by mining contextual clues between different queries of the same video, thereby enhancing the semantic relevance between the query text and the video. Then, it transfers cross-query contextual clues to a single query through knowledge distillation, thus transferring contextual clues from different queries to the semantics of a single query. Finally, it proposes an adaptive distillation mechanism that dynamically transfers contextual clues that positively enhance the semantics of the current query. This mechanism can suppress the diffusion of irrelevant clues during the clue distillation process, thereby improving the accuracy of partially relevant video retrieval. Attached Figure Description

[0015] Figure 1 This is a framework diagram of the partially related video retrieval method based on contextual clue mining of the present invention; Figure 2 This is a structural diagram of the text encoder in the partially relevant video retrieval method based on contextual clue mining of the present invention; Figure 3 This is a structural diagram of the video encoder in the partially related video retrieval method based on contextual clue mining of the present invention; Figure 4This is a framework diagram of the global clue mining module and the local clue mining module in the partially related video retrieval method based on contextual clue mining of the present invention; Figure 5 This is a graph showing the decrease in training loss of the clue mining branch in the partially relevant video retrieval method based on contextual clue mining of this invention; Figure 6 This is a graph showing the decrease in training loss of the cue distillation branch in the partially relevant video retrieval method based on contextual cue mining in this invention. Figure 7 This is a visualization of the retrieval results of the partially related video retrieval method based on contextual clue mining of the present invention and different comparison methods; Figure 8 This is a visualization of the retrieval results of the partially related video retrieval method based on contextual clue mining and different comparison methods of the present invention. Detailed Implementation

[0016] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] This invention relates to a partially relevant video retrieval method based on contextual clue mining, such as... Figure 1 As shown, the specific steps include the following: Step 1: Select a portion of the relevant video retrieval dataset and construct a corresponding query set for each query text within the dataset.

[0018] Specifically, a dataset containing video resources and corresponding query text is selected; for any query text in the dataset containing T words... Construct a query set based on other related query texts of the video V corresponding to the query text. ,in It is the first K is the number of other related query texts.

[0019] Step 2: Construct a partially relevant video retrieval network, which includes a clue mining branch and a clue distillation branch.

[0020] Specifically, such as Figure 1 As shown, the clue mining branch includes a text encoder. Video encoder The system includes a global clue mining module, a local clue mining module, a first similarity measurement module, and a text encoder. This module encodes the input query text and query set to obtain encoded query text and encoded query set; the global clue mining module and the local clue mining module calculate the global query embedding and local query embedding based on the encoded query text and encoded query set, respectively; the video encoder... This module is used to encode the input video to obtain video-level embeddings and segment-level embeddings; the first similarity measurement module is used to calculate the text-video pair similarity. Clue distillation branch includes text encoder Video encoder Second similarity measurement module; text encoder Used to encode the input query text to obtain the encoded query text, video encoder. The first module encodes the input video to obtain video-level embeddings and segment-level embeddings; the second similarity measurement module is used to calculate the text-video pair similarity.

[0021] In this invention, the text encoder of the clue mining branch Text encoder for clue distillation branches The structures are the same but the parameters are independent. The specific text encoder structure is as follows: Figure 2 As shown, the general workflow is as follows: For the input query text First, a pre-trained RoBERTa model is used to extract the query text. The features of each word are then mapped to a low-dimensional space using a fully connected neural network activated by ReLU, and learnable positional embeddings are added to obtain mapped features to preserve positional information in the text sequence. A standard Transformer encoder layer is then used to model the context of these mapped features, resulting in... Dimensional context word feature sequence Finally, in An attention module is introduced to aggregate word features and generate the final query text embedding. The specific process is as follows: (1) In the formula, It is a trainable vector. Represents the attention vector; This represents the normalization function.

[0022] In this invention, the video encoder for the clue mining branch and the video encoder of the clue distillation branch The structures are the same but the parameters are independent. The specific video encoder structure is as follows: Figure 3 As shown, the general workflow is as follows: For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches, thus constructing a multi-granular representation of the video to simultaneously capture local and global semantic information. Segment-level embeddings help the model accurately locate video segments related to the query text, while video-level embeddings are used to measure the global semantic similarity between the complete video content and the query text. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function. Then, the TC-GMMBlock module with learnable positional embeddings from Wang et al.'s 2024 arXiv paper, "GMMFormer v2: An Uncertainty-aware Framework for Partially Relevant Video Retrieval," is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows: (2) In the formula, It is a trainable vector. Represents the attention vector; In the segment-level embedding, multiple initial video frame features are first averaged and pooled. Then, a fully connected layer with ReLU activation is used to reduce the dimensionality of the pooled initial video frame features, yielding segment features. Finally, a TC-GMMBlock module with learnable positional embeddings is applied to the segment features to generate the segment-level embedding. ,in It is the number of samples. It is the feature dimension.

[0023] Step 3: Input the query text, video, and the corresponding query set constructed in Step 1 into the clue mining branch to extract global and local contextual clues, and calculate the text-video pair similarity of the clue mining branch. This specifically includes the following sub-steps: Step 3.1: Submit the query text and its corresponding query set The input clue mining branch is used to extract contextual information through the text encoder of the clue mining branch, which is represented as follows: (3) In the formula, The text encoder representing the clue mining branch, Represents the encoded query text, This represents the encoded query set; Step 3.2: Calculate the local query embedding and the global query embedding; Different queries reflect information from different moments in the video and are interconnected through contextual cues to form a complete description of the video content. The semantic complementarity of each query in the query set to the current query varies significantly. Therefore, directly using all queries in the query set as additional input and learning their alignment with the video equally does not fully leverage their semantic complementarity. Given the effectiveness of attention mechanisms in modeling different semantic regions, local contextual cues in different queries are selectively mined based on their semantic complementarity. Therefore, a local cue mining module is constructed based on the attention mechanism, such as... Figure 4 As shown, the semantic relevance between the current query text and different queries in the query set is calculated, and local contextual clues that supplement the semantics of the current query text are mined. Then, the encoded query text and local contextual clues are combined to obtain the local query embedding, as shown below: (4) In the formula, , These are two learnable linear mapping matrices; Refers to a query within a query set; Similarly, to measure the global similarity between query text and video content, a global clue mining module is constructed based on multi-head self-attention, such as... Figure 4 As shown, contextual clues reflecting the main storyline of the video are dynamically mined and aggregated. Specifically, the encoded query text and the encoded query set are stacked to form a joint representation, denoted as . Then, a global clue mining module is used to mine and aggregate global contextual clues reflecting the video level to enhance the global semantic similarity between the query text and the video. Specifically, a multi-head self-attention mechanism models the semantic relationships between different queries in different subspaces using multiple independent attention heads. Finally, an average pooling strategy is used to aggregate the intermediate results from each attention head to obtain a global query embedding reflecting the global semantics of the video, as shown below: (5) (6) In the formula, Indicates global query embedding; This indicates the average pooling operation. Indicates the first h One point of attention, The first h The projection matrix of the query, key, and value of each attention head. It is the dimension of the key. H It's about the number of heads; Step 3.3: Input the video into the clue mining branch, and pass it through the video encoder of the clue mining branch. Extracting video-level embeddings and fragment-level embedding , means as follows: (7) (8) Step 3.4: Calculate global similarity and local similarity; The first similarity logarithmic module calculates the global query embedding. and video-level embedding The cosine similarity, defined as the global similarity, is expressed as follows: (9) In the formula, Indicates global similarity. This indicates the calculation of cosine similarity. The first similarity logarithmic module uses a max-pooling strategy to calculate the local query embedding. With fragment-level embedding The similarity between them is defined as local similarity, and is expressed as follows: (10) In the formula, Indicates local similarity. Indicates the first i A video clip, Indicates max pooling; Step 3.5: Calculate the text-video pair similarity of the clue mining branch using a weighted summation method, as shown below: (11) In the formula, The text-video pair similarity represents the branching of clue mining. This represents the weight that balances global and local similarity.

[0024] Step 4: Construct the loss function for the clue mining branch and train the clue mining branch.

[0025] The clue mining branch aims to improve the semantic similarity between query text and video by mining contextual clues in the query set that enrich the semantics of the current query. This process uses a baseline loss function. The optimization loss consists of the following four parts: triplet loss function. Contrastive learning loss function Query diverse loss functions and the best matching loss function Composition, represented as follows: (12) The triplet loss function is expressed as follows: (13) In the formula, It is a boundary constant. , These respectively represent relative to video. V Negative query samples and relative to query text Negative video samples, Indicates the size of the small batch; The contrastive learning loss function is expressed as follows: (14) In the formula, Indicates a small batch relative to the video V All negative query samples, Indicates the difference between a small batch and a query. q All negative video samples; The query for diversified loss functions is as follows: (15) (16) In the formula, This indicates a controllable focusing factor. Represents the boundary factor. Indicates the scaling factor. Used to distinguish the differences between multiple query texts in the same video. This indicates the number of text queries related to a video. The optimal matching loss function is expressed as follows: (17) (18) In the formula, This indicates that each query text is constrained. Match only one video clip ; This represents the best matching scheme that maximizes the total similarity between all query texts and video segments within the same video, assuming each query text matches only one video segment.

[0026] Step 5: Distill the query text and video input clues from Step 3 into the clue distillation branch, and transfer the context clues mined in Step 3 to the clue distillation branch. Design an adaptive distillation mechanism to suppress the spread of irrelevant clues.

[0027] The goal of some related video retrieval tasks is to retrieve the video corresponding to a given query text. Therefore, during inference, the model's text input consists of only a query text. The clue mining branch, however, can use the input query set as additional information to directly mine contextual clues between different queries, enriching the semantics of the current query text. To obtain contextual clues between different queries without relying on the query set, this invention transfers the contextual clues obtained from the clue mining branch to the clue distillation branch through knowledge distillation. Specifically, this includes the following sub-steps: Step 5.1: Extract the query text from Step 3. Text encoder for input clue distillation branch Obtain the encoded global query embedding and local query embedding Simultaneously, the video from step 3 is input into the video encoder of the clue distillation branch. Get video-level embedding and fragment-level embedding , means as follows: (19) (20) (twenty one) (twenty two) The second similarity measurement module calculates the global similarity, as shown below: (twenty three) The second similarity measurement module calculates local similarity, as shown below: (twenty four) Step 5.2: Constructing the text similarity loss. First, by constraining the representational distance between the text embeddings of the cue distillation branch and the cue mining branch, the contextual cues mined in step 3 are transferred to the query text. Second, although the two branches process the same video input, the video encoding process of the cue mining branch is guided by the query semantics after fusing contextual cues, while the cue distillation branch only relies on the original query semantics, resulting in significant differences in the learning objectives of the two branches. Therefore, a visual similarity loss is constructed. The visual embedding of the clue distillation branch is imitated by the clue mining branch, thus reducing the offset between the two in the visual representation space.

[0028] Text similarity loss and visual similarity loss It is expressed as follows: (25) (26) In the formula, B Represents batch size; Step 5.3: To further align the text-video similarity distributions of the clue mining branch and the clue distillation branch in the similarity measurement stage of video retrieval, construct a distributed similarity loss. To ensure consistency in the distribution of similarity metrics between the two branches, the clue distillation branch simultaneously mimics the clue mining branch in both the feature layer and the alignment layer, thus distributing the similarity loss. It is expressed as follows: (27) In the formula, Represents cross-entropy loss, This represents the global similarity matrix between the clue mining and clue distillation branches. Represents the local similarity matrix between the clue mining and clue distillation branches; Step 5.4: During the process of mining contextual clues in the clue mining branch, some clues that are irrelevant to the current query may be learned. For example, for text in the query set... The phrase "A woman is walking along a track" can supplement the search query. The scene clue "She takes off at a fast run" is "along a track," but the verb "walking" in it semantically deviates from the action "run" emphasized in the current query, thus interfering with the ability to extract the semantic meaning of the action in the current query. It is evident that transferring all contextual clues in the query set to the clue distillation branch introduces semantic noise, affecting the retrieval performance of some relevant videos. Therefore, this invention employs an adaptive distillation mechanism to suppress the spread of irrelevant clues, transferring only some contextual clues that positively complement the semantics of the current query, thereby adaptively adjusting the knowledge distillation process. Specifically, the adaptive distillation mechanism designed in this invention uses cross-entropy loss... A learnable module is introduced to generate an adaptive mask matrix m. The loss term at each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, as shown below: (28) In the formula, Indicates position The adaptive mask value at the location; To learn the mask matrix m, the model parameters of the learning cue distillation branch are... At the same time, by minimizing the baseline loss of the clue distillation branch. gradient and optimized distribution similarity loss The difference between gradients is used to achieve adaptive adjustment of the mask. The optimization objective is expressed as: (29) In the formula, This indicates the calculation of the baseline loss function. Or optimized distribution similarity loss Regarding the gradient value of parameter 𝜃 in the clue distillation branch model.

[0029] Adaptive learning masks can effectively suppress the spread of irrelevant contextual cues while reinforcing the learning of contextual cues that significantly supplement query semantics. As the model converges, the learned mask results will tend to retain contextual cues that are positively related to query semantics.

[0030] Step 6: Construct the loss function for the cue distillation branch and train the cue distillation branch.

[0031] The constructed loss function is expressed as follows: (30) In the formula, , It's a hyperparameter.

[0032] Step 7: Since the query set cannot be obtained directly, the query text to be retrieved is input into the trained cue distillation branch to perform partial related video retrieval.

[0033] Example 1 This embodiment provides a method for retrieving partially relevant videos based on contextual clue mining, specifically including the following steps: Step 1: Select a portion of the relevant video retrieval dataset and construct a corresponding query set for each query text within the dataset; Step 2: Construct a partially relevant video retrieval network, which includes a clue mining branch and a clue distillation branch; Step 3: Input the query text, video, and the corresponding query set constructed in Step 1 into the clue mining branch to extract global and local context clues, and calculate the text-video pair similarity of the clue mining branch; Step 4: Construct the loss function for the clue mining branch and train the clue mining branch; Step 5: Distill the query text and video input clues from Step 3 into the clue distillation branch, and transfer the context clues mined in Step 3 to the clue distillation branch. Design an adaptive distillation mechanism to suppress the spread of irrelevant clues. Step 6: Construct the loss function for the cue distillation branch and train the cue distillation branch; Step 7: Input the query text to be retrieved into the trained cue distillation branch to perform partial related video retrieval.

[0034] Example 2 Building upon Example 1, the public dataset Charades-STA was selected, containing 6,670 videos and 16,128 text descriptions. The average video length is approximately 30 seconds. Each video contains an average of 2.4 moments with corresponding text descriptions. For any query text in the dataset containing T words... Construct a query set based on other related query texts of the video V corresponding to the query text. ,in It is the first K is the number of other related query texts.

[0035] Example 3 Based on Example 2, in step 2, the clue mining branch includes a text encoder. Video encoder The system includes a global clue mining module, a local clue mining module, a first similarity measurement module, and a text encoder. This module encodes the input query text and query set to obtain encoded query text and encoded query set; the global clue mining module and the local clue mining module calculate the global query embedding and local query embedding based on the encoded query text and encoded query set, respectively; the video encoder... This module is used to encode the input video to obtain video-level embeddings and segment-level embeddings; the first similarity measurement module is used to calculate the text-video pair similarity. Clue distillation branch includes text encoder Video encoder Second similarity measurement module; text encoder Used to encode the input query text to obtain the encoded query text, video encoder. The first module encodes the input video to obtain video-level embeddings and segment-level embeddings; the second similarity measurement module is used to calculate the text-video pair similarity.

[0036] Text encoder for clue mining branch Text encoder for clue distillation branches The structures are the same, and the workflow is as follows: For the input query text First, a pre-trained RoBERTa model is used to extract the query text. The features of each word are extracted, resulting in 1024-dimensional features. Subsequently, a ReLU-activated fully connected neural network maps each word's features to a low-dimensional space, and learnable positional embeddings are added to obtain the mapped features. Then, a standard Transformer encoder layer is used to perform contextual modeling on the mapped features, yielding... Dimensional context word feature sequence Finally, in An attention module is introduced to aggregate word features and generate the final query text embedding. The specific process is as follows: (1) In the formula, It is a trainable vector. Represents the attention vector; Represents the normalization function; Video encoder for clue mining branch and the video encoder of the clue distillation branch The structures are the same, and the workflow is as follows: For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with learnable positional embeddings is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows: (2) In the formula, It is a trainable vector. Represents the attention vector; In the segment-level embedding, the initial video frame features are first subjected to average pooling; then, a fully connected layer with ReLU activation is used to reduce the dimensionality of the pooled initial video frame features to obtain segment features; finally, the TC-GMMBlock module with learnable position embedding is applied to the segment features to generate the segment-level embedding. ,in It is the number of samples. It is the feature dimension.

[0037] Example 4 Based on Example 3, step 3 specifically includes the following steps: Step 3.1: Submit the query text and its corresponding query set The input clue mining branch is used to extract contextual information through the text encoder of the clue mining branch, which is represented as follows: (3) In the formula, The text encoder representing the clue mining branch, Represents the encoded query text, This represents the encoded query set; Step 3.2: Calculate the local query embedding and the global query embedding; A local clue mining module is constructed based on an attention mechanism. It calculates the semantic relevance between the current query text and different queries in the query set, and mines local contextual clues that supplement the semantics of the current query text. Then, the encoded query text and local contextual clues are combined to obtain the local query embedding, as shown below: (4) In the formula, , These are two learnable linear mapping matrices; Refers to a query within a query set; A global clue mining module is constructed based on multi-head self-attention, which stacks the encoded query text and the encoded query set to form a joint representation denoted as . Then, the global clue mining module mines and aggregates global contextual clues reflecting the video level to obtain the global query embedding, as shown below: (5) (6) In the formula, Indicates global query embedding; This indicates the average pooling operation. Indicates the first h One point of attention, The first h The projection matrix of the query, key, and value of each attention head. It is the dimension of the key. H It's about the number of heads; Step 3.3: Input the video into the clue mining branch, and pass it through the video encoder of the clue mining branch. Extracting video-level embeddings and fragment-level embedding , means as follows: (7) (8) Step 3.4: Calculate global similarity and local similarity; The first similarity logarithmic module calculates the global query embedding. and video-level embedding The cosine similarity, defined as the global similarity, is expressed as follows: (9) In the formula, Indicates global similarity. This indicates the calculation of cosine similarity. The first similarity logarithmic module uses a max-pooling strategy to calculate the local query embedding. With fragment-level embedding The similarity between them is defined as local similarity, and is expressed as follows: (10) In the formula, Indicates local similarity. Indicates the first i A video clip, Indicates max pooling; Step 3.5: Calculate the text-video pair similarity of the clue mining branch using a weighted summation method, as shown below: (11) In the formula, The text-video pair similarity represents the branching of clue mining. This embodiment represents the weights that balance global and local similarity. ; The loss function for the clue mining branch constructed in step 4 is expressed as follows: (12) In the formula, Represents the baseline loss function. Represents the triplet loss function. This represents the contrastive learning loss function. This indicates a query for a diversified loss function. This represents the optimal matching loss function; The triplet loss function is expressed as follows: (13) In the formula, It is a boundary constant. , These respectively represent relative to video. V Negative query samples and relative to query text Negative video samples, Indicates the size of the small batch; The contrastive learning loss function is expressed as follows: (14) In the formula, Indicates a small batch relative to the video V All negative query samples, Indicates the difference between a small batch and a query. q All negative video samples; The query for diversified loss functions is as follows: (15) (16) In the formula, This indicates a controllable focusing factor. Represents the boundary factor. Indicates the scaling factor. Used to distinguish the differences between multiple query texts in the same video. This indicates the number of text queries related to a video. The optimal matching loss function is expressed as follows: (17) (18) In the formula, This indicates that each query text is constrained. Match only one video clip ; This represents the best matching scheme that maximizes the total similarity between all query texts and video segments within the same video, assuming each query text matches only one video segment.

[0038] Example 5 Based on Example 4, step 5 specifically includes the following sub-steps: Step 5.1: Extract the query text from Step 3. Text encoder for input clue distillation branch Obtain the encoded global query embedding and local query embedding Simultaneously, the video from step 3 is input into the video encoder of the clue distillation branch. Get video-level embedding and fragment-level embedding , means as follows: (19) (20) (twenty one) (twenty two) The second similarity measurement module calculates the global similarity, as shown below: (twenty three) The second similarity measurement module calculates local similarity, as shown below: (twenty four) Step 5.2: Constructing the text similarity loss. By constraining the representation distance between the text embeddings of the cue distillation branch and the text embeddings of the cue mining branch, the contextual cues mined in step 3 are transferred to the query text; simultaneously, a visual similarity loss is constructed. The visual embedding of clue distillation branches mimics the clue mining branches; Text similarity loss and visual similarity loss It is expressed as follows: (25) (26) In the formula, B Represents batch size; Step 5.3: Construct the distribution similarity loss To ensure consistency in the distribution of similarity metrics between the two branches, the clue distillation branch simultaneously mimics the clue mining branch in both the feature layer and the alignment layer, thus distributing the similarity loss. It is expressed as follows: (27) In the formula, Represents cross-entropy loss, This represents the global similarity matrix between the clue mining and clue distillation branches. Represents the local similarity matrix between the clue mining and clue distillation branches; Step 5.4: Design an adaptive distillation mechanism: through cross-entropy loss A learnable module is introduced to generate an adaptive mask matrix m. The loss term at each position of the cross-entropy loss is weighted to generate an optimized distribution similarity loss, as shown below: (28) In the formula, Indicates position The adaptive mask value at the location; To learn the mask matrix m, the model parameters of the learning cue distillation branch are... At the same time, by minimizing the baseline loss of the clue distillation branch. gradient and optimized distribution similarity loss The difference between gradients is used to achieve adaptive adjustment of the mask. The optimization objective is expressed as: (29) In the formula, This indicates the calculation of the baseline loss function. Or optimized distribution similarity loss Regarding the gradient value of parameter 𝜃 in the clue distillation branch model.

[0039] Example 6 Based on Example 5, the loss function constructed in step 6 is expressed as follows: (30) In the formula, , It's a hyperparameter. =0.2, =1.0; Simulation Experiment This simulation experiment, based on Example 6, was conducted on a single RTX 4090 GPU. The model was trained and tested using Python 3.8 and the PyTorch 2.0.0 framework, with computational acceleration supported by CUDA 11.8 and cuDNN 8.0. Regarding experimental parameters, the model learning rate (lr) was set to 0.00025.

[0040] Using a ranking-based evaluation index R@K, K∈{1,5,10,100}, as shown in Equation (31), it measures the top K The percentage of videos successfully retrieved from each search result, corresponding to the query, is expressed as a percentage. A higher R@K value indicates better method performance. Furthermore, SumR is used to measure different... K The sum of R@K values ​​is used to evaluate overall retrieval performance.

[0041] (31) To verify the effectiveness of the method of the present invention, it is compared with (short) video retrieval methods RIVRL, CLIP4Clip and Cap4Video, video corpus time-based retrieval methods XML, ReLoCLNet and CONQUER, and some related video retrieval methods MS-SL, PEAN, T-D3N, GMM and GMM-V2.

[0042] Experimental results: The retrieval performance of the partially related video retrieval method proposed in this invention compared with different comparison methods is as follows: Figure 3As shown in Table 1, the (short) video retrieval method performs the worst because it measures the complete relevance between text and video content, which limits its ability to retrieve partially relevant videos. For example, the RIVRL method has a SUMR of only 54.3%, the lowest among all methods. The purpose of video corpus time-based retrieval methods is to retrieve time information related to the query text, which helps to learn the partial relevance between text and video to some extent. Compared with mainstream partially relevant video retrieval methods, the method of this invention achieves state-of-the-art performance, with a SUMR index 2.3% higher than the best partially relevant video retrieval method.

[0043] Table 1

[0044] The above experimental analysis shows that the global contextual clues mined in the method of this invention help discover potential information related to the preceding and following parts of the video content, thus measuring global text-video similarity. Local contextual clues can supplement the video details missing in a single query text, thereby promoting segment-level local alignment by the model. This demonstrates the effectiveness and innovation of the method of this invention in partially relevant video retrieval from the perspective of contextual clue mining. Furthermore, by using an adaptive distillation mechanism to transfer some contextual clues that positively enhance the semantics of the current query, rather than all clues, this effectively avoids redundant information that may be introduced during distillation and solves the problem that it is impossible to obtain all query texts for a video in actual retrieval, thus improving the robustness of the retrieval.

[0045] Tables 2 and 3 respectively show the retrieval performance graphs of different global and local contextual clue mining methods in this invention. Feature concatenation fuses text semantics by concatenating multiple query text features along the same dimension. Max pooling helps the model fuse key semantic information from different queries by extracting the maximum value within a local region of query text features. The attention method dynamically mines semantic associations between queries by assigning a weight coefficient between 0 and 1 to each word of each query text.

[0046] Table 2

[0047] Table 3

[0048] As shown in Tables 2 and 3, the attention method can selectively aggregate textual contextual clues that positively complement the semantics of the query text and have finer granularity, thus better alleviating the semantic asymmetry problem between the two modalities and achieving the best performance. Taking local contextual clue mining as an example, the SUMR index of the attention method is 31.7% and 16.5% higher than that of feature concatenation and max pooling methods, respectively. In contrast, the method of simply concatenating different query features treats the degree of complementarity of each query to the semantics of the current query equally, but not every word in every query has a positive semantic complementarity to the current query, so its performance is the worst. Although the max pooling method can highlight some semantically relevant clues, it cannot dynamically pay attention to the importance of semantics at different positions, and may miss some key contextual clue information. In addition, both methods may introduce other irrelevant or redundant information, affecting the overall performance of the model.

[0049] Figure 5 and Figure 6 The decreasing trends of training loss for the cue mining branch and the cue distillation branch in the method of this invention are visualized separately. The figures show that the cue mining branch tends to converge smoothly after 800 iterations, while the cue distillation branch's loss value stabilizes after 2000 iterations. Experimental results demonstrate that the model in the method of this invention can converge effectively without overfitting or getting trapped in local optima.

[0050] Figure 7 and Figure 8 This diagram presents a visual comparison of the retrieval results of the method of this invention with other representative video retrieval methods, such as MS-SL and GMM-V2. The videos marked with boxes in the diagram are the actual videos corresponding to the query text. Figure 7 While the MS-SL method successfully captured the action semantics "takes" in the query text, it overlooked the key semantic "blanket," leading to its top-3 search results being similar but irrelevant videos. The GMM-V2 method, through its excellent video segment modeling capabilities, mitigated this problem to some extent, but still exhibited retrieval bias. In contrast, the method of this invention effectively distinguishes these mismatched videos by mining contextual clues such as "door," "closet," and "light" implicit in the query set. The semantics of the query text after incorporating these contextual clues can more accurately describe the video content, thereby precisely retrieving the target video. Figure 8 Similar phenomena were observed, further verifying the effectiveness of the contextual clue mining method proposed in this invention in improving the semantic similarity between query text and video.

Claims

1. A partial-relevant video retrieval method based on context cue mining, characterized in that, Specifically comprising the following steps: Step 1: Selecting a partial correlation video retrieval data set, constructing a corresponding query set for each query text in the data set; Step 2: Constructing a partial correlation video retrieval network, the partial correlation video retrieval network comprising a clue mining branch and a clue distillation branch; Step 3: Inputting the query text, video and corresponding query set constructed in step 1 into the clue mining branch to extract global and local context clues, and calculating the text-video pair similarity of the clue mining branch; Step 4: Constructing a loss function for the clue mining branch, and training the clue mining branch; Step 5: Inputting the query text and video in step 3 into the clue distillation branch, transferring the context clues mined in step 3 to the clue distillation branch, and designing an adaptive distillation mechanism to suppress the diffusion of irrelevant clues; Step 6: Constructing a loss function for the clue distillation branch, and training the clue distillation branch; Step 7: Inputting the query text to be retrieved into the trained clue distillation branch for partial correlation video retrieval.

2. The method for partial relevant video retrieval based on context cues mining according to claim 1, wherein, Step 1 is specifically: A dataset comprising video resources and with corresponding query texts is selected; for any one query text comprising T words in the dataset , a query set is constructed based on other related query texts of the video V corresponding to the query text , wherein is the th other related query text, and K is the number of other related query texts.

3. The method for partial relevant video retrieval based on context cue mining according to claim 2, wherein, In step 2, the clue mining branch includes a text encoder , a video encoder , a global clue mining module, a local clue mining module, and a first similarity measurement module, the text encoder is configured to encode the input query text and the query set to obtain encoded query text and encoded query set; the global clue mining module and the local clue mining module are respectively configured to calculate global query embedding and local query embedding according to the encoded query text and the encoded query set; the video encoder is configured to encode the input video to obtain video-level embedding and segment-level embedding; and the first similarity measurement module is configured to calculate text-video pair similarity. The clue distillation branch comprises a text encoder , a video encoder and a second similarity measure module; the text encoder is configured to encode input query text to obtain encoded query text, the video encoder is configured to encode input video to obtain video-level embedding and segment-level embedding; and the second similarity measure module is configured to calculate text-video pair similarity.

4. The partial correlation video retrieval method based on context clue mining according to claim 3, characterized in that, The text encoder of the lead mining branch The text encoder of the lead distillation branch The structure is the same, and the workflow is as follows: For the input query text , first, the pre-trained RoBERTa model is used to extract the features of each word in the query text ; then, the features of each word are mapped to a low-dimensional space by a fully connected layer neural network with ReLU activation and added with a learnable position embedding to obtain the mapped features, and then a standard Transformer encoder layer is used to model the context of the mapped features to obtain a sequence of context word features in -dimensional space; finally, an attention module is introduced on to aggregate the word features to generate the final query text embedding , the specific process is as follows: （1） wherein is a trainable vector, denotes an attention vector; denotes a normalization function; Video encoder of the clue mining branch Video encoder of the clue distillation branch The structure is the same, and the workflow is as follows: For including Uncropped video frames First, a pre-trained 2D convolutional neural network is used to extract I3D features from each video frame to obtain initial video frame features. Then, the initial video frame features are modeled into video-level embeddings and segment-level embeddings through two parallel branches. In the video-level embedding, the initial video frame features are first reduced in dimensionality using a fully connected layer with a ReLU activation function, and then a TC-GMMBlock module with learnable positional embeddings is applied to obtain contextual features. Finally, in Generate video embeddings using attention modules , means as follows: （2） wherein is a trainable vector, denotes an attention vector; In the segment-level embedding, firstly, average pooling is performed on the initial video frame features; then, a fully connected layer with ReLU activation function is used to reduce the dimension of the pooled initial video frame features to obtain segment features; finally, the TC-GMMBlock module with learnable position embedding is applied to the segment features to generate the segment-level embedding wherein is the number of samples, is the feature dimension.

5. The method of claim 4, wherein, Step 3 specifically comprises the following steps: Step 3.1: Query text and its corresponding query set Input the clue mining branch, and the context information is extracted by the text encoder of the clue mining branch, which is represented as: （3） In the formula, a text encoder representing a clue mining branch, an encoded query text, an encoded query set; Step 3.2: Calculate the local query embedding and the global query embedding; A local clue mining module is constructed based on an attention mechanism to calculate the semantic relevance between the current query text and different queries in the query set, mine local context clues that supplement the semantics of the current query text, and then combine the encoded query text and the local context clues to obtain a local query embedding, which is represented as follows: （4） wherein , are two learnable linear mapping matrices; denotes one query in the query set; A global clue mining module is constructed based on multi-head self-attention, and the encoded query text and the encoded query set are stacked to form a joint representation denoted as , and then the global clue mining module is used to mine and aggregate the global context clues reflecting the video level to obtain a global query embedding, which is denoted as follows: （5） （6） wherein, denotes a global query embedding; denotes an average pooling operation, denotes the h th attention head, are the projection matrices for the query, key and value of the h th attention head, respectively, is the dimension of the key, H is the number of attention heads; Step 3.3: Video input cue mining branch, by cue mining branch video encoder extracting video-level embeddings and segment-level embeddings , is represented as follows: （7） （8） Step 3.4: Calculate the global similarity and the local similarity; The first similarity pair module computes global query embeddings and video-level embeddings The cosine similarity of the global query embedding and the video-level embedding, defined as global similarity, is represented as follows: （9） In the formula, represents the global similarity, represents the cosine similarity; The first similarity pair module employs a max-pooling strategy to compute local query embeddings with segment-level embeddings The similarity with the segment-level embedding, defined as a local similarity, is denoted as follows: （10） wherein denotes a local similarity, denotes the i video segment, denotes max pooling; Step 3.5: Calculate the text-video pair similarity of the clue mining branch by weighted summation, which is represented as follows: （11） wherein, represents the text-video pair similarity of the clue mining branch, represents the weight balancing global and local similarities.

6. The method for partial relevant video retrieval based on context cue mining according to claim 1, wherein, The loss function of the clue mining branch constructed in step 4 is represented as follows: （12） wherein denotes the baseline loss function, denotes the triple loss function, denotes the contrastive learning loss function, denotes the query diversification loss function, denotes the best match loss function; The triplet loss function is represented as follows: （13） wherein, is a boundary constant, , respectively represent negative query samples with respect to videos V and negative video samples with respect to query texts , denotes a mini-batch size; The contrastive learning loss function is represented as follows: （14） wherein represents all negative query samples in the mini-batch with respect to the video V represents all negative query samples in the mini-batch with respect to the video represents all negative video samples in the mini-batch with respect to the query q represents all negative video samples in the mini-batch with respect to the query The query diversification loss function is represented as follows: （15） （16） wherein, denotes a controllable focusing factor, denotes a boundary factor, denotes a scaling factor, for distinguishing the difference between multiple query texts in the same video, denotes the number of text queries related to a video; The best match loss function is represented as follows: （17） （18） wherein represents to constrain each query text to match only one video segment ; represents to let the best matching scheme of all query texts in the same video with the highest total similarity of video segments in the case of each query text matching only one video segment.

7. The method according to claim 6, wherein, Step 5 specifically comprises the following sub-steps: Step 5.1: encode the query text from step 3 into a text embedding Text encoder of the input cue distillation branch to obtain a global query embedding and a local query embedding Step 5.2: encode the video input cue distillation branch from step 3 into a video embedding to obtain a video-level embedding and a segment-level embedding , denoted as follows: （19） （20） （21） （22） The second similarity measurement module calculates the global similarity, which is represented as follows: （23） The second similarity measurement module calculates the local similarity, which is represented as follows: （24） Step 5.2: Constructing the text similarity loss by constraining the representation distance between the text embeddings of the cue distillation branch and the cue mining branch text embeddings to transfer the mined contextual cues from the query text in step 3; At the same time, a visual similarity loss is constructed The guiding clue distillation branch imitates the visual embedding of the clue mining branch. the text similarity loss and the visual similarity loss is represented as follows: （25） （26） In the formula, B represents the batch size; Step 5.3: Constructing Distribution Similarity Loss , constraining the consistency of the two branch similarity measure distributions, so that the clue distillation branch simultaneously imitates the clue mining branch at both the feature layer and the alignment layer, the distribution similarity loss is represented as follows: （27） wherein denotes the cross-entropy loss, denotes the global similarity matrix of the clue mining and clue distillation branches, denotes the local similarity matrix of the clue mining and clue distillation branches; Step 5.4: Designing an adaptive distillation mechanism: By weighting the loss terms for each position of the cross-entropy loss Lce with a learnable mask matrix m, the distribution similarity loss Ldis is generated as follows: The learnable mask matrix m is introduced as follows: （28） In the formula, indicates the adaptive mask value at the position ​ For learning the mask matrix m, the model parameters of the learning clue distillation branch are learned at the same time, by minimizing the difference between the gradient of the reference loss of the clue distillation branch and the gradient of the optimized distribution similarity loss of the mask, the adaptive adjustment of the mask is realized, and the optimization target is represented as: （29） In the formula, represents the reference loss function or the optimized distribution similarity loss The gradient value of the clue distillation branch model parameter θ.

8. The method according to claim 7, wherein, The loss function constructed in step 6 is represented as follows: （30） wherein , are hyperparameters.

Images