A method and system for improving video moment retrieval performance based on counterfactual learning

Abstract

The application discloses a method and system for improving video time retrieval performance based on counterfactual learning, inputting a video and a corresponding text query; extracting video features and text features; interacting and aligning the video features and the text features to obtain an encoder embedding; inputting the obtained encoder embedding into a counterfactual sample synthesis module to obtain a counterfactual encoder content embedding and a counterfactual encoder position embedding; obtaining a decoder output by using a Transformer decoder module; inputting the decoder output into a causality module to calculate a causal effect of the encoder embedding, and obtaining a causal representation. The causal table and the decoder output obtained from the encoder embedding are subjected to time sequence prediction to obtain a set prediction; the predicted result and the real label are subjected to supervised training, the influence of false correlation is effectively reduced, the generalization performance of the model on different distribution data is enhanced, and the model can more accurately locate a time segment related to a natural language query in the video.

Description

Technical Field

[0001] This invention relates to a method and system for improving video time-of-view retrieval performance based on counterfactual learning, which is a method for enhancing the generalization ability of the DETR model in video time-of-view retrieval tasks, and belongs to the field of computer vision technology. Background Technology

[0002] With the development of streaming media, accurately finding relevant content from massive amounts of video is not easy. Therefore, technologies such as video summarization, video retrieval, and video generation have been widely explored in recent years. Video temporal localization is a key task, aiming to locate the temporal boundaries of relevant video segments in undressed videos based on a given natural language query, greatly reducing the time users spend searching for videos.

[0003] Existing methods for solving video time-of-view retrieval tasks are mainly divided into two categories: proposal-based and proposal-free. Proposal-based methods rely on techniques such as sliding windows and 2D time graphs to generate candidate segments, and then rank them according to their relevance to the query. Proposal-free methods, on the other hand, directly regress the target boundary to avoid the complex process of generating candidate segments. Among them, the DETR-based method treats time-of-view retrieval as an end-to-end ensemble prediction problem. By introducing a learnable query to predict the boundary of a specific instance, it has advantages such as no need for complex manual design, low computational cost, small storage requirements, and the ability to effectively capture scene context information and inter-frame temporal relationships, making it a popular choice.

[0004] However, DETR-based methods face challenges from data bias in practical applications. Data bias stems from the subjectivity of data annotation and the distributional differences between the training and test sets, making the model prone to relying on spurious relevance rather than learning the true semantic relationships between videos and text. For example, in different settings of the Charades dataset, the timestamp distribution for the "open" action query is biased, causing existing DETR models to accurately locate target segments on the in-distribution (IID) test set but to make incorrect predictions on the out-of-distribution (OOD) test set, severely limiting the model's generalization ability.

[0005] To address data bias, existing research has employed methods such as data augmentation, data reweighting, and data sampling. However, these methods typically require expanding the data sample, inevitably introducing a large amount of redundant and noisy samples. Meanwhile, some methods utilizing causal inference, such as DCM and IVG, while introducing confounding variables and performing backdoor adjustments to estimate causal effects, struggle to handle unobservable confounding factors due to the complex multimodal interactions in video time-lapse retrieval. Applying counterfactual inference to DETR also presents challenges, as its query-based ensemble prediction paradigm aligns the prediction span with the actual time-lapse through binary matching, making it difficult to directly estimate classifier-level causal effects. Summary of the Invention

[0006] Purpose of the invention: This invention aims to solve the problem that the generalization ability of existing DETR-based video time-of-view retrieval models is limited due to data bias. It provides a method based on counterfactual learning, which effectively reduces the impact of spurious relevance, enhances the generalization performance of the model on data with different distributions, and enables the model to more accurately locate time segments in the video that are relevant to natural language queries.

[0007] Technical Solution: A method for improving video time-of-view retrieval performance based on counterfactual learning, used to enhance the generalization ability of DETR-based video time-of-view retrieval models. Specifically, it includes the following steps:

[0008] Step 1: Retrieve the model input video and corresponding text query at the video moment.

[0009] Step 2: Extract video and text features using a pre-trained feature extraction module. Video features can be extracted using convolutional neural networks or multimodal models at a set video frame sampling rate. Text features can be extracted using the word embedding tool GloVe.

[0010] Step 3: Use two multilayer perceptrons to map video features and text features to the same feature dimension.

[0011] Step 4: Use the multimodal interaction module to interactively align video and text features to obtain multimodal features. The multimodal interaction module includes a multi-layer Transformer encoder, whose input is video and text features, and whose output is the encoder embedding, i.e., the multimodal features.

[0012] Step 5: Input the obtained encoder embedding into the counterfactual sample synthesis module to obtain the counterfactual encoder content embedding and the counterfactual encoder position embedding.

[0013] Step 6: Next, input the counterfactual encoder content embedding, the counterfactual encoder position embedding, and the encoder embedding together into the Transformer decoder module to obtain the decoder output.

[0014] Step 7: The decoder output input causal module, obtained from the counterfactual encoder content embedding and the counterfactual encoder position embedding, calculates the causal effect of the encoder embedding to obtain a causal representation.

[0015] Step 8: Use the prediction module to perform temporal prediction on the decoder output obtained from the causal table and encoder embedding to obtain set prediction.

[0016] Step 9: Perform supervised training using the predicted results and the actual labels. The final loss form is:

[0017]

[0018] in and It's about balancing hyperparameters. For counterfactual losses, and These are the loss functions for the auxiliary and basic branches, respectively, and are related to... They are in the same form.

[0019] The process by which the causal module calculates the causal effect embedded in the encoder and obtains the causal representation is as follows:

[0020] First, the causal effect of the encoder embedding on the decoder output is defined as the difference between the final queries generated from factual and counterfactual inputs; the causal effects of content and location interventions are respectively expressed as... , ;

[0021] Next, a simple gating fusion strategy is used to calculate the weight w, as shown in the formula:

[0022] ;

[0023] in, It is a sigmoid function; For connection operations; by fusing causal effects through learned weights, the causal effects are first transformed. , Then a unified causal representation can be obtained. ,in It is the hyperbolic tangent function.

[0024] A system for improving video moment retrieval performance based on counterfactual learning includes a feature extraction module, a multimodal interaction module, a counterfactual sample synthesis module, a Transformer decoder module, a causal module, a prediction module, and a counterfactual learning module.

[0025] The feature extraction module retrieves video and corresponding text queries from the input video time-lapse model, extracting video and text features. Video features can be extracted using convolutional neural networks or multimodal models at a set video frame sampling rate. Text features can be extracted using the word embedding tool GloVe. Two multilayer perceptrons are used to map video and text features to the same feature dimension.

[0026] The multimodal interaction module performs interactive alignment of video and text features to obtain multimodal features. The multimodal interaction module includes a multi-layered Transformer encoder, whose inputs are video and text features, and whose output is the encoder embedding, i.e., the multimodal features.

[0027] The counterfactual sample synthesis module takes an encoder embedding as input to obtain the counterfactual encoder content embedding and the counterfactual encoder position embedding.

[0028] In the Transformer decoder module, the counterfactual encoder content embedding, the counterfactual encoder position embedding, and the encoder embedding are input together into the Transformer decoder module to obtain the decoder output.

[0029] In the causal module, the decoder output obtained from the counterfactual encoder content embedding and the counterfactual encoder position embedding is input to the causal module to calculate the causal effect of the encoder embedding and obtain a causal characterization.

[0030] The prediction module performs temporal prediction on the causal table and the decoder output obtained by encoder embedding to obtain set prediction.

[0031] The counterfactual learning module is trained under supervision using predicted results and real labels.

[0032] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for improving video moment retrieval performance based on counterfactual learning as described above.

[0033] A computer-readable storage medium storing a computer program that performs the method described above for improving video moment retrieval performance based on counterfactual learning.

[0034] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0035] Improve generalization performance: Effectively reduce the impact of false relevance caused by data bias, significantly improve the video moment retrieval performance of the DETR-based video moment retrieval model on in-distribution (IID) and out-of-distribution (OOD) test sets, enhance the model's generalization ability, and enable it to more accurately locate relevant video segments in different data distribution scenarios.

[0036] Model independence: The method of improving video time-of-view retrieval performance based on counterfactual learning is model independent and can be flexibly integrated into any DETR-based video time-of-view retrieval model. It provides a general debiased learning solution for existing models without the need for large-scale modifications to the model structure, thus reducing application costs. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the method principle of an embodiment of the present invention;

[0038] Figure 2 This is a cause-effect structure diagram of an embodiment of the present invention;

[0039] Figure 3 This is a comparison chart of experimental results. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0041] A method and system for improving video time-of-view retrieval performance based on counterfactual learning are presented to enhance the generalization ability of DETR-based video time-of-view retrieval models. Counterfactual samples are generated by intervening in the multimodal embeddings of the encoder, a causal module is designed to separate causal embeddings, and a counterfactual learning objective is introduced to make the model focus on robust video content relevant to the text query, thereby achieving more accurate time-of-view retrieval.

[0042] like Figure 1 As shown, the method for improving video moment retrieval performance based on counterfactual learning includes feature extraction, multimodal interaction, counterfactual sample synthesis, causal module, and counterfactual learning.

[0043] The system for improving video moment retrieval performance based on counterfactual learning includes a feature extraction module, a multimodal interaction module, a counterfactual sample synthesis module, a Transformer decoder module, a causal module, a prediction module, and a counterfactual learning module.

[0044] like Figure 2As shown, a causal structure diagram is presented. The cross-modal alignment process in the DETR decoder is captured by modeling the causal dependencies between data bias, multimodal features, moment queries, decoder output, and predictions. Data bias acts as a confounding factor, establishing spurious associations between multimodal features and predictions.

[0045] Methods for improving video moment retrieval performance based on counterfactual learning include:

[0046] Input a video and a corresponding text query, and use a pre-trained feature extraction module to extract video features. and text features .in, Indicates the number of video frames. This represents the number of text tokens. Video features are typically extracted using convolutional neural networks (VGG, I3D, C3D) or multimodal models (CLIP) at a specific video frame sampling rate. Text features can be extracted using the word embedding tool GloVe. Two multilayer perceptrons are used to map video and text features to the same feature dimension.

[0047]

[0048] MLP(.) stands for Multilayer Perceptron.

[0049] A multimodal interaction module is used to fuse video and text features. The extracted video and text features are used as input to the multimodal interaction module to obtain the encoder embedding, calculated as follows:

[0050]

[0051] in, B represents the encoder embedding; D represents the number of samples in the batch training; L represents the number of tokens; and Encoder represents the encoder module of the Transformer.

[0052] Counterfactual sample synthesis. After obtaining the encoder embedding, the counterfactual encoder content is embedded. and counterfact encoder position embedding Interventions were conducted to investigate their causal effects on the decoder output.

[0053] Content embedding intervention is to embed the original content Replace with counterfactual embedding This is used to intervene in the interaction between query and content information; the counterfactual encoder content embedding, counterfactual encoder position embedding, and encoder embedding are input together into the Transformer decoder module to obtain the decoder output. .

[0054] Interventions can take many forms, such as replacing the original content embedding with learnable tokens, fixed constant values, or values ​​sampled from a Gaussian distribution. Specifically, the intervention can be represented as follows:

[0055] ;

[0056] in Let P be a random variable drawn from a Bernoulli distribution, and let X represent the filling matrix. Let M be the Hadamard product, and M represent the encoder embedding. A certain proportion of features P are randomly selected for intervention to balance preserving useful information and mitigating bias. Based on this, the counterfactual output of the decoder under content intervention is obtained:

[0057] .

[0058] in, This indicates a learnable query token. Indicates the number of query tokens; This represents the output of the decoder; Decoder represents the decoder module of the Transformer.

[0059] Location embedding intervention is similar to content embedding intervention, involving the application of location embedding. Intervention, where 'do' represents the intervention operation. This primarily involves randomly shuffling the original position embeddings to explore their impact on the decoder output and mitigate bias. This leads to the counterfactual output of the decoder under positional intervention:

[0060] .

[0061] Causal module construction. The causal module calculates the causal effect of the encoder embedding by comparing the output embeddings obtained by the decoder from factual and counterfactual inputs.

[0062] The decoder output input causal module, obtained from the counterfactual encoder content embedding and the counterfactual encoder position embedding, calculates the causal effect of the encoder embedding to obtain a causal representation. .

[0063] Using the prediction module to characterize causality Decoder output obtained by encoder embedding Perform time series forecasting to obtain set forecast Y.

[0064] First, the causal effect of the encoder embedding on the decoder output is defined as the difference between the final query generated from factual and counterfactual inputs. Therefore, the causal effects of content and location interventions are expressed as follows: , .

[0065] Next, a simple gating fusion strategy is used to calculate the weight w, as shown in the formula:

[0066] ;

[0067] in, It is a sigmoid function; This is a connection operation. The causal effects are fused using learned weights, and the causal effects are first transformed. , Then a unified causal representation can be obtained. ,in It is the hyperbolic tangent function.

[0068] Once the decoder output is obtained, predictions can be made using two perceptrons as prediction heads.

[0069]

[0070] in, Indicates the start and end timestamps of the prediction; Indicates the classification probability of foreground / background; Represents a sigmoid function; FC(.) represents a fully connected layer;

[0071] Counterfactual learning objective. By introducing counterfactual learning loss, the difference between the causal representation's predictions and the actual moments is measured. After obtaining the model's output, the optimal binary match between the predictions and the actual moments is first found using the Hungarian algorithm.

[0072]

[0073] in It is a causal representation The prediction set It is a collection of real events. It is a matching cost function. Based on optimal allocation. Calculate counterfactual learning loss ;

[0074] in , For cross-entropy loss, It is L1 loss. It is a loss for GIoU. , and It's about balancing hyperparameters;

[0075] Due to the instability of bipartite graph matching, an auxiliary training branch is constructed. First, a corresponding time query is constructed for each ground truth label, so that the set prediction of the causal representation directly matches the ground truth, bypassing the bipartite graph matching process. The rest of the process is consistent with the main branch.

[0076] Finally, the overall loss function is

[0077]

[0078] in and It's about balancing hyperparameters. For counterfactual losses, and These are the loss functions for the auxiliary and basic branches, respectively, and are related to... They are in the same form.

[0079] This invention is adaptable to existing DETR-based video temporal retrieval methods. In specific implementation, this invention is trained on two public datasets, Charades-CD and ActivityNet-CD. The Transformer encoder and decoder used in this invention have 3 layers, 256 hidden dimensions, and 8 attention heads. The AdamW optimizer used has 200 training epochs, a weight decay of 1e-4, and a learning rate of 2e-4. The training batch sizes for Charades-CD and ActivityNet-CD datasets are set to 16 and 64, respectively. The loss balancing hyperparameters are set accordingly. and The values ​​are all set to 1. This invention uses pre-trained VGG, C3D, and SlowFast+CLIP models as video encoders to extract video features, and uses the Glove model and CLIP text model to extract 300-d and 512-d text features, respectively, for the text encoder.

[0080] exist Figure 3 The qualitative results on the OOD test set of the Charades-CD dataset are visualized. Video-text pairs exhibiting bias distributions are selected; for example, queries containing the action "eat" are typically paired with actions occurring in the first half of the video. Compared to the influential video temporal retrieval model OD-DETR, the method of this embodiment shows more accurate localization results, demonstrating enhanced generalization ability and better robustness to data bias.

Claims

1. A method for improving video moment retrieval performance based on counterfactual learning, characterized in that, To improve the generalization ability of the DETR-based video time-of-view retrieval model, the following steps are included: Step 1: Retrieve the model input video and corresponding text query at each video moment; Step 2: Extract video and text features using a pre-trained feature extraction module; Step 3: Use two multilayer perceptrons to map video features and text features to the same feature dimension; Step 4: Use the multimodal interaction module to interactively align video features and text features to obtain the encoder embedding; Step 5: Input the obtained encoder embedding into the counterfactual sample synthesis module to obtain the counterfactual encoder content embedding and the counterfactual encoder position embedding; After obtaining the encoder embedding, embed the counterfactual encoder content. and counterfact encoder position embedding Interventions were conducted to investigate their causal effects on the decoder output; Content embedding intervention is to embed the original content Replace with counterfactual embedding To intervene in the interaction between query and content information; the form of intervention is ； in It is a random variable drawn from a Bernoulli distribution, with probability parameter . , Represents a filling matrix. For Hadama accumulation, Indicates encoder embedding; randomly selects a set ratio. The features are intervened to balance retaining useful information and mitigating bias; based on this, the counterfactual output of the decoder under content intervention is obtained: in, This indicates a learnable query token. Indicates the number of query tokens; This represents the output of the decoder; Decoder represents the decoder module of the Transformer. Location embedding intervention is similar to content embedding intervention, involving the application of location embedding. Intervention, among which This indicates an intervention operation; the impact of randomly shuffling the original position embeddings on the decoder output is explored, and bias is mitigated. This leads to the counterfactual output of the decoder under positional intervention: ； Step 6: Input the counterfactual encoder content embedding, counterfactual encoder position embedding, and encoder embedding together into the Transformer decoder module to obtain the decoder output; Step 7: The decoder output input causal module, obtained from the counterfactual encoder content embedding and the counterfactual encoder position embedding, calculates the causal effect of the encoder embedding to obtain a causal representation; Step 8: Use the prediction module to perform temporal prediction on the causal table and the decoder output obtained by the encoder embedding to obtain set prediction; Step 9: Supervised training is performed using the predicted results and the actual labels; the final loss form is: in and It's about balancing hyperparameters. For counterfactual losses, and These are the loss functions for the auxiliary and basic branches, respectively, and are related to... They are in the same form.

2. The method for improving video time-of-view retrieval performance based on counterfactual learning according to claim 1, characterized in that, In step 2, video features and text features are extracted using a pre-trained feature extraction module; for video features, a convolutional neural network or a multimodal model is used to extract them according to a set video frame sampling rate; for text features, the word embedding tool GloVe is used to extract them.

3. The method for improving video time-of-view retrieval performance based on counterfactual learning according to claim 1, characterized in that, In step 4, the video features and text features are interactively aligned using a multimodal interaction module to obtain multimodal features. The multimodal interaction module includes a multi-layer Transformer encoder, whose input is video and text features, and whose output is the encoder embedding, i.e., multimodal features.

4. The method for improving video time-of-view retrieval performance based on counterfactual learning according to claim 1, characterized in that, The process by which the causal module calculates the causal effect embedded in the encoder and obtains the causal representation is as follows: First, the causal effect of the encoder embedding on the decoder output is defined as the difference between the final queries generated from factual and counterfactual inputs; the causal effects of content and location interventions are respectively expressed as... , ; Next, a simple gating fusion strategy is used to calculate the weights. The formula is ； in, It is a sigmoid function; For connection operations; by fusing causal effects through learned weights, the causal effects are first transformed. , Then a unified causal representation can be obtained. ,in It is the hyperbolic tangent function.

5. The method for improving video time-of-view retrieval performance based on counterfactual learning according to claim 1, characterized in that, After obtaining the decoder output, prediction is performed using two perceptrons as prediction heads. in, Indicates the start and end timestamps of the prediction; Indicates the classification probability of foreground / background; Represents a sigmoid function; Indicates a fully connected layer; The counterfactual learning objective, by introducing a counterfactual learning loss, measures the difference between the causal representation's predictions and the actual moments. After obtaining the model's output, the Hungarian algorithm is first used to find the optimal binary match between the predictions and the actual moments. in It is a causal representation The prediction set It is a collection of real events. It is a matching cost function; based on optimal allocation Calculate counterfactual learning loss ; in , For cross-entropy loss, It is L1 loss. It is a loss for GIoU. , and It's about balancing hyperparameters; Due to the instability of bipartite graph matching, an auxiliary training branch is constructed. First, a corresponding time query is constructed for each real label, so that the set prediction of the causal representation directly matches the real, bypassing the bipartite graph matching process. The rest of the process is consistent with the main branch.

6. A system implemented using the method for improving video time-of-view retrieval performance based on counterfactual learning as described in claim 1, characterized in that, It includes a feature extraction module, a multimodal interaction module, a counterfactual sample synthesis module, a Transformer decoder module, a causal module, a prediction module, and a counterfactual learning module; The feature extraction module extracts video features and text features from the video and corresponding text query in the input video time-retrieval model. The multimodal interaction module performs interactive alignment of video features and text features to obtain multimodal features; The counterfactual sample synthesis module takes an encoder embedding as input to obtain the counterfactual encoder content embedding and the counterfactual encoder position embedding. In the Transformer decoder module, the counterfactual encoder content embedding, the counterfactual encoder position embedding, and the encoder embedding are input together into the Transformer decoder module to obtain the decoder output; In the causal module, the decoder output obtained from the counterfactual encoder content embedding and the counterfactual encoder position embedding is input to the causal module to calculate the causal effect of the encoder embedding and obtain a causal characterization; The prediction module performs temporal prediction on the causal table and the decoder output obtained by encoder embedding to obtain set prediction; The counterfactual learning module is trained under supervision using predicted results and real labels.

7. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for improving video time-of-view retrieval performance based on counterfactual learning as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method for improving video time-of-view retrieval performance based on counterfactual learning as described in any one of claims 1-5.

Images