Combined video retrieval method and system based on hierarchical uncertainty perception disambiguation
By employing a global semantic reference resolution mechanism and atomic-level uncertainty modeling, the problems of ambiguous object reference and insufficient attention to detail in combined video retrieval are solved, thereby improving the accuracy and recall of multimodal queries.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-03-31
AI Technical Summary
Existing combined video retrieval methods struggle to effectively capture the semantic differences between video and text modalities, leading to ambiguity in the referent of modified objects and limited attention to detailed semantics, which affects the model's multimodal query understanding.
A global semantic reference resolution mechanism and an atomic uncertainty modeling strategy were designed. Through video-text cross-modal interaction, semantic overlap regions were mined. Gaussian probability embedding was combined to model text uncertainty, enhancing attention to key details. A multi-level semantic alignment structure was used to achieve accurate alignment between multimodal queries and target videos.
It enables precise location and detail focus of modified objects in combined video retrieval, improves the model's multimodal query understanding ability, and increases retrieval recall.
Smart Images
Figure CN121765115A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a combined video retrieval method and system based on hierarchical uncertainty perception disambiguation, belonging to the technical field of multimodal retrieval. Background Technology
[0002] In recent years, video and text have gradually become important carriers of information representation, storage, and dissemination in the Internet age. The scale of multimedia data constituted by these two has grown exponentially, and their forms have become increasingly complex and diverse. Traditional video retrieval methods (such as "searching for videos by video" based solely on video content, or "searching for videos by text" based solely on text) are no longer sufficient to meet users' increasingly complex and diverse refined query needs. Therefore, the Composed Video Retrieval (CVR) paradigm has emerged. Composed video retrieval aims to combine user-provided reference videos and modified text to accurately understand the user's search intent and efficiently retrieve target videos that meet the user's complex needs from massive video libraries. However, because video data is semantically rich and has high information density, while text is usually ambiguous and has limited information density, how to effectively capture the semantic differences between video and text modalities and accurately express the user's query intent has become a key challenge in current combined video retrieval research. Previous methods have ignored the information density differences between video and text modalities. Compared to text modalities, video modalities often contain richer semantics. Furthermore, this phenomenon leads to the following two problems, affecting the performance of CVR models.
[0003] 1) Ambiguous referent of the object to be modified. The object referred to by "it" in the modification text is not explicitly stated. It is necessary to consider the context of the modification text and infer from "swallow" that "it" in the modification text refers to the "bird" in the video. Unfortunately, this is not an isolated case. Most of the modification texts in the current CVR dataset use pronouns such as "it" and "they" to refer to the object to be modified, or simply lack a subject to modify. This increases the difficulty for the model to determine the visual object to be modified and is detrimental to the model's understanding of multimodal queries.
[0004] 2) Limited attention to semantic details. In the reference video, most areas are occupied by "trees" and "sky," while the detail that actually needs modification is the "bird," which has a relatively small visual footprint. Objects with a small visual footprint but playing a crucial role in the modification requirements are called "critical details." However, existing CVR models often give equal attention to different visual regions, which may lead to insufficient attention being paid to "critical details," resulting in biased combined features. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a combined video retrieval method based on Leveled Uncertainty Perception Disambiguation CVR-Net (LUPD-Net), which aims to improve multimodal query understanding by leveraging the information density differences between video and text.
[0006] The challenging combined video retrieval task that this invention aims to solve is: how to retrieve target videos from a video list that meet the requirements of multimodal combined video queries. Let... As a group A set of triples, in which These represent the reference video, modified text, and target video, respectively. Essentially, the goal of this invention is to learn a metric space in which multimodal queries... and the corresponding target video The embedded features should be as close as possible, which can be formally expressed as: ,in The embedding function to be learned represents the multimodal query and the target video.
[0007] To address this, this invention first designs a global semantic reference resolution mechanism module. This module aims to model the uncertainty of the modified object at the global level through overlapping semantics in global intermodal interactions, indirectly disambiguating the modified object referred to by textual pronouns and resolving the problem of ambiguous object reference. Secondly, this invention designs an atomic-level uncertainty modeling strategy. This strategy aims to model fine-grained cross-modal semantics through atomic-level cross-modal interactions, supplementing global semantics and improving the detail attention capability of the CVR model, thus solving the problem of limited detail semantic attention. Finally, this invention designs a multi-level semantic alignment structure module. This module adaptively aligns multimodal queries and target videos at the global-atomic level through learnable similarity bias.
[0008] Terminology Explanation: Q-Former is a deep learning model architecture for vision-language tasks such as visual question answering and image retrieval. It's an improved Transformer model specifically designed to enhance the interaction between images and text and improve performance across modal tasks. Its core idea is to capture the relationships between images and text through a custom query mechanism.
[0009] The technical solution of this invention is as follows: A combined video retrieval method based on hierarchical uncertainty-aware disambiguation, implemented through a trained combined video retrieval model, includes: The model understands each element of a given triple, including the reference video, target video, and modified text, and generates corresponding embedding features for each. It then combines the reference video and modified text to generate multimodal query combined features. A triple refers to the metadata generated during the training of the combined video retrieval model, including the reference video, modified text, and target video. The reference video and target video use the same video encoder to generate embedding features, while the modified text uses a corresponding text encoder. A multimodal query refers to the query input of the combined video retrieval model, including the reference video and modified text. Cross-modal semantic alignment is performed between multimodal query combined features and target video features to achieve combined video retrieval.
[0010] As a further preferred approach, the elements of a given triple are understood and corresponding embedded features are generated. Simultaneously, the reference video and modified text are combined and understood to generate multimodal query combined features; including: Global semantic reference resolution mechanism: By leveraging the cross-modal global interaction structure between video and text, we can uncover semantically overlapping areas and assist in locating specific video objects from the overall semantic level by addressing ambiguous references in the text. Atomic-level uncertainty modeling: Introducing an atomic-level interaction mechanism, combined with Gaussian probability embedding to model uncertain semantics in text, and refining the focus on key details in the video.
[0011] As a further preferred approach, cross-modal semantic alignment is performed between the multimodal query combined features and the target video features; including: By employing a holistic-atomic dual-layer semantic representation approach, a fusion structure based on learnable similarity bias is constructed to achieve multi-level semantic adaptive alignment from text to video.
[0012] As a further preferred solution, a global semantic reference resolution mechanism includes: First, overall visual and textual features are extracted within a unified semantic space; specifically, a pre-trained Q-Former is used to extract tokens that modify the text and features sampled from reference videos. The frame, the formula is as follows: (1); in, , Atomic tokens representing the reference video and the modified text, respectively. It is the number of sampling frames. L This refers to the number of queries that can be learned in Q-Former. Indicates the atomic embedding dimension; and These represent the visual encoder and the text segmenter, respectively. , These refer to the reference video and the revised text, respectively. Next, the atomic tokens are globally averaged and projected to generate a global token for the reference video and modified text. ,in, Indicates the overall embedding dimension; The probability distribution of the modified text is modeled using a Gaussian probability distribution, denoted as . ,in, and Let represent the mean and variance of the probability of modifying the text, respectively, as shown in the following formulas: (2); in, , It includes a linear layer, a sigmoid activation function, and a normalized layer. This results in a single linear layer; This represents the cross-attention mechanism used to calculate the mean. This represents the cross-attention mechanism used to calculate variance. The overall token is for reference videos and modified text; Subsequently, reparameterization techniques were employed from the distribution The formula for a random variable with independent and identically distributed iid samples is as follows: (3); in, This represents the text probability embedding, where 𝑈 represents the number of samples. Represents a standard normal distribution; For the u-th probability embedding of the sample, , These represent the mean and variance of the probability of modifying the text, respectively. iid represents the probability embedding of samples taken from a standard normal distribution; iid is an abbreviation for independent and identically distributed sampling. The overall combined token is constructed from both probability distribution and original features; including: First, a method for constructing a total combined token based on probability distribution; including: Using a multilayer perceptron (MLP) to learn the text probability embedding for the nth sample With reference overall token Combined weights The formula is: (4); in, ; Subsequently, Perform the block partitioning operation and record the resulting weights as follows: ; Then, the weights are respectively and Aggregate to obtain the nth overall probabilistic combination token. The formula is: (5); in, ; For the original features Apply the same combination operation to obtain the overall combined token. , This refers to linear fusion operation; As a further preferred option, atomic-level uncertainty modeling strategies include: First, construct a Gaussian probability distribution for the reference video, denoted as . ,in, Let represent the mean and variance of the probability of the reference video, respectively, and the formula is: (6); in, and Having the same architecture, This indicates the obtained reference video and modified text atomic tokens, and ; Subsequently, from the distribution For each token, independent and identically distributed random variables are sampled to model the visual detail embedding of each token, as shown below: (7); in, Indicates visual detail embedding. For the l-th probability embedding of the sample, The probability embedding represents the sampled probability from the standard normal distribution; Using visual detail embedding As a query embedding, and embedding visual details With Modify Text Token Sequence We can use Q-Former for fine-grained interaction, as described below: (8); in, Tokens representing atomic uncertainty. This represents a text segmenter; fine-grained interactions between the original reference video atomic tokens and the modified text token sequence; and alignment of the feature dimensions of the atomic and overall tokens through projection, as shown below: (9); in, Indicates atomic combination tokens, It refers to the linear mapping layer.
[0013] As a further preferred option, a multi-level semantic alignment structure includes: First, obtain the overall and atomic target tokens, and expand their channel dimensions to match the dimensions of the overall and atomic constituent tokens, denoted as follows: ; Subsequently, a similarity deviation calculation function is defined. for: (10); in, These represent the nth token of the overall combined token and the target token, respectively; Then, the similarity bias term is incorporated into the original similarity calculation as a weighting factor for the overall similarity; thus, the learnable similarity function... The definition is as follows: (11); Similarly, atomic-level similarity is obtained, denoted as... ; This refers to a dimension of 1; Subsequently, a batch-based classification loss was employed. Combining overall and atomic similarity, the formula is: (12); Where 𝐵 represents the batch size and 𝜏 represents the temperature coefficient. These represent the nth overall combined token and the target token in the batch, respectively; set up Let represent the similarity distribution of the ith overall combined token in the batch, where The similarity between the nth overall combined token and the nth overall target token is defined as: (13); Similarly, the similarity distribution of the nth overall target token in the batch is obtained, denoted as... And the similarity distribution of atomic combination tokens and target tokens in the batch, denoted as , respectively. Next, we define distribution regularization. as follows: (14); Finally, the following optimization function for LUPD-NET is obtained. : (15); Here, Θ is the LUPD-NET parameter to be learned, and 𝜅 is the trade-off hyperparameter.
[0014] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a combined video retrieval method based on hierarchical uncertainty-aware disambiguation.
[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a combined video retrieval method based on hierarchical uncertainty-aware disambiguation.
[0016] A combined video retrieval system based on hierarchical uncertainty-aware disambiguation includes: The multimodal query combination feature generation module is configured to: understand each element of a given triple, including the reference video, the target video, and the modified text, and generate corresponding embedding features respectively; and combine and understand the reference video and the modified text to generate multimodal query combination features. The multi-level semantic alignment structure module is configured to perform cross-modal semantic alignment between multimodal query combined features and target video features to achieve combined video retrieval.
[0017] A further preferred embodiment of the multimodal query combination feature generation module includes a global semantic reference resolution mechanism module and an atomic-level uncertainty modeling module; The global semantic reference resolution mechanism module is configured to: mine semantically overlapping areas through the video-text cross-modal global interaction structure, and assist in locating specific video objects from the overall semantic level by locating fuzzy references in the text; The atomic uncertainty modeling module is configured to: introduce an atomic interaction mechanism, combine Gaussian probability embedding to model uncertain semantics in the text, and refine the focus on key details in the video.
[0018] The beneficial effects of this invention are as follows: 1. This invention designs a novel combined video retrieval model, LUPD-NET, which is the first CVR framework that utilizes the information density differences between video and text to improve multimodal query understanding.
[0019] 2. The LUPD-NET proposed in this invention can achieve object disambiguation by using overlapping semantics of global intermodal interactions and semantics of atomic-level cross-modal interactions, and improve attention to detailed semantics to achieve accurate combinatorial learning. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the overall process of a combined video retrieval method based on hierarchical uncertainty perception disambiguation according to the present invention. Detailed Implementation
[0021] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.
[0022] Example 1 A combined video retrieval method based on hierarchical uncertainty-aware disambiguation is implemented through a trained combined video retrieval model, such as... Figure 1 As shown, it includes: The model understands each element of a given triple, including the reference video, target video, and modified text, and generates corresponding embedding features for each. It then combines the reference video and modified text to generate multimodal query combined features. A triple refers to the metadata generated during the training of the combined video retrieval model, including the reference video, modified text, and target video. The reference video and target video use the same video encoder to generate embedding features, while the modified text uses a corresponding text encoder. A multimodal query refers to the query input of the combined video retrieval model, including the reference video and modified text. By performing cross-modal semantic alignment between multimodal query combination features and target video features, the combined video retrieval model designed in this invention can effectively evaluate the semantic similarity between "multimodal query combination features and target video features" and achieve combined video retrieval.
[0023] Example 2 The difference between the combined video retrieval method based on hierarchical uncertainty perception disambiguation as described in Example 1 and the method described in Example 1 is as follows: The system interprets each element of a given triplet and generates corresponding embedded features. Simultaneously, it combines and interprets reference videos and modified text to generate multimodal query composite features; including: Global semantic reference resolution mechanism: By leveraging the cross-modal global interaction structure between video and text, it mines semantically overlapping areas and assists in locating specific video objects from the overall semantic level by addressing ambiguous references (such as "it" and "they") in the text; thereby enhancing the understanding of user intent.
[0024] Atomic-level uncertainty modeling: This approach introduces an atomic-level interaction mechanism, combining Gaussian probability embedding to model uncertain semantics in text, refining the focus on key details in videos (such as small objects in the scene and local actions). This addresses the problem of global generalization in traditional methods.
[0025] Perform cross-modal semantic alignment between multimodal query combined features and target video features; including: Multi-level semantic alignment structure: By employing a holistic-atomic two-layer semantic representation approach, a fusion structure based on learnable similarity bias is constructed to achieve multi-level semantic adaptive alignment from text to video. This improves the accuracy and robustness of cross-modal semantic representation.
[0026] Global semantic reference resolution mechanism; including: In CVR tasks, edited text often contains ambiguous pronouns referring to the object being edited. For example, the pronoun "it" appears in the edited text. It is difficult to determine which object "it" in the reference video refers to based solely on the edited text. This invention proposes to address this problem by capturing overlapping semantics from a holistic perspective through cross-modal interaction. Through this interaction between the edited text and the reference video, the model can infer that "it" refers to a "bird," rather than a "tree" or "sky." Based on this intuition, this invention designs a holistic pronoun disambiguation module. This module indirectly resolves pronoun ambiguity by utilizing cross-modal interaction, thereby alleviating the problem of unclear referents of the edited subject. The following is a detailed description of this module:
[0027] To facilitate cross-modal interaction, firstly, overall visual and textual features are extracted within a unified semantic space; specifically, a pre-trained Q-Former is used to extract tokens for modifying text and features sampled from reference videos. The frame, the formula is as follows: (1); in, , Atomic tokens representing the reference video and the modified text, respectively. It is the number of sampling frames. L This refers to the number of queries that can be learned in Q-Former. Indicates the atomic embedding dimension; and These represent the visual encoder and the text segmenter, respectively. , These refer to the reference video and the revised text, respectively. Next, in order to obtain a holistic representation of the video and text, the atomic tokens are globally averaged and projected to generate a holistic token for the reference video and modified text. ,in, Indicates the overall embedding dimension; Inspired by the application of probability distributions in image / video and text representation, this invention constructs a probability distribution for modified text to enhance the understanding of ambiguous modification semantics. Furthermore, it utilizes a cross-attention mechanism to mine overlapping semantics between different modalities. This method can establish semantic associations between textual pronouns and their corresponding modified objects, and indirectly achieve pronoun disambiguation.
[0028] Specifically, the probability distribution of the modified text is modeled using the widely used Gaussian probability distribution, denoted as... ,in, and Let represent the mean and variance of the probability of modifying the text, respectively, as shown in the following formulas: (2); in, , It includes a linear layer, a sigmoid activation function, and a normalized layer. It is a single linear layer; so far, we have extracted the overlapping semantics between the modified text and the reference video through cross-attention and embedded these semantics into the Gaussian probability distribution of the modified text. This represents the cross-attention mechanism used to calculate the mean. This represents the cross-attention mechanism used to calculate variance. The overall token is for reference videos and modified text;
[0029] Subsequently, due to the interruption of gradient propagation caused by sample randomness, a reparameterization technique was employed to obtain gradients from the distribution. The formula for a random variable with independent and identically distributed iid samples is as follows: (3); in, This represents the text probability embedding, where 𝑈 represents the number of samples. Represents a standard normal distribution; For the u-th probability embedding of the sample, , These represent the mean and variance of the probability of modifying the text, respectively. iid represents the probability embedding of samples taken from a standard normal distribution; iid is an abbreviation for independent and identically distributed sampling. To facilitate subsequent comprehensive learning, this invention constructs an overall combined token from both probability distribution and original features; including: First, a method for constructing a total combined token based on probability distribution; including: Using a multilayer perceptron (MLP) to learn the text probability embedding for the nth sample With reference overall token Combined weights The formula is: (4); in, ; Subsequently, Perform the block partitioning operation and record the resulting weights as follows: ; Then, the weights are respectively and Aggregate to obtain the nth overall probabilistic combination token. The formula is: (5); in, ; It is worth noting that, in addition to overlapping semantics, the reference video and modified text also contain different semantics, which typically encapsulate key modification information (such as color changes and position modifications). Although overlapping semantics have been mined using probability distributions, in order to extract different semantics and ensure the semantic integrity of multimodal queries, this invention further refines the original features... Apply the same combination operation to obtain the overall combined token. , This refers to the linear fusion operation; combining two features.
[0030] The ultimate goal is to "assist in locating specific video objects from ambiguous references in the text at the overall semantic level," and the final referential result is the overall combined token. .
[0031] Atomic-level uncertainty modeling strategies include: As previously mentioned, this invention recognizes that some objects in a video frame may occupy a small area but are important targets for modification; these can be termed "critical details." Existing CVR models typically allocate equal attention to all visual regions, which may lead to insufficient attention to these critical detail objects, thus affecting the effectiveness of composition learning. To enhance the model's attention to critical details, this invention introduces atomic uncertainty modeling. It aims to identify critical detail semantics from an atomic perspective through cross-modal interaction and uncertainty modeling, serving as a supplement to the overall semantics. This improves the LUPD-NET model's ability to focus on details and effectively solves the problem of insufficient focus on detailed semantics. This module will be described in detail below:
[0032] Since the reference video contains numerous details, only the modified object can be considered a "critical detail." Therefore, to enhance the model's attention to detail, it is necessary to identify "critical details" among the many details. Specifically, an atomic-level uncertainty modeling method is introduced for visual detail embedding. Similar to probabilistic embedding, firstly, a Gaussian probability distribution is constructed for the reference video, denoted as . ,in, Let represent the mean and variance of the probability of the reference video, respectively, and the formula is:
[0033] (6); in, and Having the same architecture, This indicates the obtained reference video and modified text atomic tokens, and ; Subsequently, since each reference atomic token may contain key modification detail semantics, from the distribution For each token, independent and identically distributed random variables are sampled to model the visual detail embedding of each token, as shown below: (7); in, Indicates visual detail embedding. For the l-th probability embedding of the sample, The probability embedding represents the sampled probability from the standard normal distribution; To ensure that the generated composite tokens can simultaneously capture both the overall and atomic semantics of multimodal queries, this invention constructs atomic composite tokens corresponding to the overall composite token construction. Unlike overall composite tokens, atomic composite tokens focus more on fine-grained modification details. Therefore, this invention utilizes Q-Former's fine-grained attention mechanism to establish a detailed-level association between visual and textual modalities, thereby achieving accurate detail extraction and multimodal composition.
[0034] Using visual detail embedding As a query embedding, and embedding visual details With Modify Text Token Sequence We can use Q-Former for fine-grained interaction, as described below: (8); in, Tokens representing atomic uncertainty. This represents a text segmenter; similar to whole composition, to preserve the different semantics of the original atomic tokens, fine-grained interactions are performed between the original reference video atomic tokens and the modified text token sequence; furthermore, to facilitate subsequent whole-to-atom alignment, the feature dimensions of the atomic and whole tokens are aligned through projection, represented as: (9); in, Indicates atomic combination tokens, It refers to the linear mapping layer.
[0035] Multi-level semantic alignment structures include: Due to the inherent uncertainty of the previously sampled independent and identically distributed random variables, directly calculating the feature similarity between the multimodal query and the target video may introduce semantic bias. To address this issue, this invention designs a global-to-local alignment, introducing a learnable bias term into the similarity calculation. This eliminates any semantic bias introduced by the probability distribution and guides the tokens composed of the global and local components to adaptively align with the target video.
[0036] Specifically, first, the same process as in formula (1) is used to obtain the overall and atomic target tokens, and their channel dimensions are expanded to match the dimensions of the overall and atomic constituent tokens, denoted as follows: ; Subsequently, taking overall similarity calculation as an example, a similarity deviation calculation function is defined. for: (10); in, These represent the nth token of the overall combined token and the target token, respectively; Then, the similarity bias term is incorporated into the original similarity calculation as a weighting factor for the overall similarity; thus, the learnable similarity function... The definition is as follows: (11); Similarly, atomic-level similarity is obtained, denoted as... ; This refers to a dimension of 1; Subsequently, in order to simultaneously bring the combined tokens of the whole and atoms closer to the corresponding target video, a batch-based classification loss is employed. Combining overall and atomic similarity, the formula is: (12); Where 𝐵 represents the batch size and 𝜏 represents the temperature coefficient. These represent the nth overall combined token and the target token in the batch, respectively; Furthermore, since the global and atomic tokens are modeled separately, their semantic distributions may be inconsistent. To ensure that the atomic semantics captured at a fine-grained level do not conflict with the global semantics, distribution regularization is introduced to promote distribution consistency across different semantic levels. Specifically, let... Let represent the similarity distribution of the ith overall combined token in the batch, where The similarity between the nth overall combined token and the nth overall target token is defined as:
[0037] (13); Similarly, the similarity distribution of the nth overall target token in the batch is obtained, denoted as... And the similarity distribution of atomic combination tokens and target tokens in the batch, denoted as , respectively. Next, we define distribution regularization. as follows: (14); Finally, the following optimization function for LUPD-NET is obtained. : (15); Here, Θ is the LUPD-NET parameter to be learned, and 𝜅 is the trade-off hyperparameter.
[0038] Cross-modal semantic alignment of multimodal query combined features with target video features enables combined video retrieval, a method for video retrieval based on combined features. In this process, the goal of combined video retrieval (CVR) is to find the best-matching result for the target video using multimodal input (i.e., a combination of reference video and modified text).
[0039] The proposed combined video retrieval method based on hierarchical uncertainty-aware disambiguation, abbreviated as LUPD-NET, achieves the optimal retrieval recall rate when compared on the globally recognized combined video retrieval datasets WebVid-CoVR and EgoCVR. To clearly compare the advantages and disadvantages of related combined video retrieval methods, Tables 1-2 show the retrieval recall rate comparison results of each method on the WebVid-CoVR and EgoCVR datasets.
[0040] Table 1. Recall rate on the WebVid-CoVR dataset; Table 2. Retrieval recall on the EgoCVR dataset;
[0041] Tables 1 and 2 show the retrieval recall rates of various models on the WebVid-CoVR and EgoCVR datasets. Below is an explanation of each model in the tables and the meaning of the relevant parameters.
[0042] Specifically, CLIP is a contrastive learning-based visual-language model capable of handling image-text matching tasks within a shared embedding space. BLIP is another visual-language model that improves performance on multimodal tasks by optimizing the alignment between images and text. CoVR (AAAI'24) was the first model to propose this task, demonstrating strong cross-modal retrieval capabilities by combining visual and textual features. CoVR_enrich (CVPR'24) is an enhanced version of COVR, proposing additional image annotations as an aid to improve retrieval accuracy. CoVR-2 (TPAMI'24) is the second-generation version of COVR, further refining the feature alignment strategy and further enhancing performance. FDCA (ICLR'25) introduces feature discriminant and cross-attention mechanisms, making the model more robust in cross-modal tasks. LUPD-NET is the model of this invention.
[0043] In Tables 1 and 2, the parameter "R@k" represents recall (Recall at k), which evaluates the proportion of correct results found by the model in the first k search results. Specifically, "R@1" indicates whether the correct answer can be found in the first result, "R@5" indicates whether the correct answer can be found in the first 5 results, and "R@10" and "R@50" correspond to the performance in the first 10 and first 50 results, respectively. "Avg." is the average of these recall values, reflecting the overall performance of the model across multiple k values.
[0044] The following direct conclusions can be drawn from Tables 1 and 2. First, the LUPD-NET model demonstrates the strongest performance across multiple recall metrics, especially surpassing other models in recall on the WebVid-CoVR and EgoCVR datasets. Second, the CONR series and FDCA models also show strong advantages in recall, particularly on the EgoCVR dataset, where CONR-2 and FDCA excel in local recall. While CLIP and BLIP are more basic models, they remain stable in some tasks, and EgoVLPv2 exhibits good performance on the EgoCVR dataset, especially in global recall. In summary, the LUPD-NET model, with its powerful cross-modal retrieval capabilities, clearly outperforms existing mainstream methods.
[0045] Example 3 A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a combined video retrieval method based on hierarchical uncertainty perception disambiguation as described in Embodiment 1 or 2.
[0046] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a combined video retrieval method based on hierarchical uncertainty perception disambiguation as described in Embodiment 1 or 2.
[0047] Example 5 A combined video retrieval system based on hierarchical uncertainty-aware disambiguation includes: The multimodal query combination feature generation module is configured to: understand each element of a given triple, including the reference video, the target video, and the modified text, and generate corresponding embedding features respectively; and combine and understand the reference video and the modified text to generate multimodal query combination features. The multi-level semantic alignment structure module is configured to perform cross-modal semantic alignment between multimodal query combined features and target video features to achieve combined video retrieval.
[0048] The multimodal query combined feature generation module includes a global semantic reference resolution mechanism module and an atomic uncertainty modeling module; The global semantic reference resolution mechanism module is configured to: mine semantically overlapping areas through the video-text cross-modal global interaction structure, and assist in locating specific video objects from the overall semantic level by locating fuzzy references in the text; The atomic uncertainty modeling module is configured to: introduce an atomic interaction mechanism, combine Gaussian probability embedding to model uncertain semantics in the text, and refine the focus on key details in the video.
Claims
1. A combined video retrieval method based on hierarchical uncertainty perception disambiguation, characterized in that, This is achieved through a pre-trained combined video retrieval model, including: The model understands each element of a given triple, including the reference video, target video, and modified text, and generates corresponding embedding features for each. It then combines the reference video and modified text to generate multimodal query combined features. A triple refers to the metadata generated during the training of the combined video retrieval model, including the reference video, modified text, and target video. The reference video and target video use the same video encoder to generate embedding features, while the modified text uses a corresponding text encoder. A multimodal query refers to the query input of the combined video retrieval model, including the reference video and modified text. Cross-modal semantic alignment is performed between multimodal query combined features and target video features to achieve combined video retrieval.
2. The combined video retrieval method based on hierarchical uncertainty perception disambiguation according to claim 1, characterized in that, The system interprets each element of a given triple and generates corresponding embedded features. Simultaneously, it combines and interprets reference videos and modified text to generate multimodal query composite features; including: Global semantic reference resolution mechanism: By leveraging the cross-modal global interaction structure between video and text, we can uncover semantically overlapping areas and assist in locating specific video objects from the overall semantic level by addressing ambiguous references in the text. Atomic-level uncertainty modeling: Introducing an atomic-level interaction mechanism, combined with Gaussian probability embedding to model uncertain semantics in text, and refining the focus on key details in the video.
3. The combined video retrieval method based on hierarchical uncertainty perception disambiguation according to claim 1, characterized in that, Perform cross-modal semantic alignment between multimodal query combined features and target video features; including: By employing a holistic-atomic dual-layer semantic representation approach, a fusion structure based on learnable similarity bias is constructed to achieve multi-level semantic adaptive alignment from text to video.
4. The combined video retrieval method based on hierarchical uncertainty perception disambiguation according to claim 2, characterized in that, Global semantic reference resolution mechanism; including: First, overall visual and textual features are extracted within a unified semantic space; specifically, a pre-trained Q-Former is used to extract tokens that modify the text and features sampled from reference videos. The frame, the formula is as follows: （1）； in, , Atomic tokens representing the reference video and the modified text, respectively. It is the number of sampling frames. L This refers to the number of queries that can be learned in Q-Former. Indicates the atomic embedding dimension; and These represent the visual encoder and the text segmenter, respectively. , These refer to the reference video and the revised text, respectively. Next, the atomic tokens are globally averaged and projected to generate a global token for the reference video and modified text. ,in, Indicates the overall embedding dimension; The probability distribution of the modified text is modeled using a Gaussian probability distribution, denoted as . ,in, and Let represent the mean and variance of the probability of modifying the text, respectively, as shown in the following formulas: （2）； in, , It includes a linear layer, a sigmoid activation function, and a normalized layer. This results in a single linear layer; This represents the cross-attention mechanism used to calculate the mean. This represents the cross-attention mechanism used to calculate variance. The overall token is for reference videos and modified text; Subsequently, reparameterization techniques were employed from the distribution The formula for a random variable with independent and identically distributed iid samples is as follows: （3）； in, This represents the text probability embedding, where 𝑈 represents the number of samples. Represents a standard normal distribution; For the u-th probability embedding of the sample, , These represent the mean and variance of the probability of modifying the text, respectively. iid represents the probability embedding of samples taken from a standard normal distribution; iid is an abbreviation for independent and identically distributed sampling. The overall combined token is constructed from both probability distribution and original features; including: First, a method for constructing a total combined token based on probability distribution; including: Using a multilayer perceptron (MLP) to learn the text probability embedding for the nth sample With reference overall token Combined weights The formula is: （4）； in, ; Subsequently, Perform the block partitioning operation and record the resulting weights as follows: ; Then, the weights are respectively and Aggregate to obtain the nth overall probabilistic combination token. The formula is: （5）； in, ; For the original features Apply the same combination operation to obtain the overall combined token. , This refers to linear fusion operations.
5. A combined video retrieval method based on hierarchical uncertainty perception disambiguation according to claim 2, characterized in that, Atomic-level uncertainty modeling strategies include: First, construct a Gaussian probability distribution for the reference video, denoted as . ,in, Let represent the mean and variance of the probability of the reference video, respectively, and the formula is: （6）； in, and Having the same architecture, This indicates the obtained reference video and modified text atomic tokens, and ; Subsequently, from the distribution For each token, independent and identically distributed random variables are sampled to model the visual detail embedding of each token, as shown below: （7）； in, Indicates visual detail embedding. For the l-th probability embedding of the sample, The probability embedding represents the sampled probability from the standard normal distribution; Using visual detail embedding As a query embedding, and embedding visual details With Modify Text Token Sequence We can use Q-Former for fine-grained interaction, as described below: （8）； in, Tokens representing atomic uncertainty. This represents a text segmenter; fine-grained interactions between the original reference video atomic tokens and the modified text token sequence; and alignment of the feature dimensions of the atomic and overall tokens through projection, represented as follows: （9）； in, Indicates atomic combination tokens, It refers to the linear mapping layer.
6. A combined video retrieval method based on hierarchical uncertainty perception disambiguation according to any one of claims 1-5, characterized in that, Multi-level semantic alignment structures include: First, obtain the overall and atomic target tokens, and expand their channel dimensions to match the dimensions of the overall and atomic constituent tokens, denoted as follows: ; Subsequently, a similarity deviation calculation function is defined. for: （10）； in, These represent the nth token of the overall combined token and the target token, respectively; Then, the similarity bias term is incorporated into the original similarity calculation as a weighting factor for the overall similarity; thus, the learnable similarity function... The definition is as follows: （11）； Similarly, atomic-level similarity is obtained, denoted as... ; This refers to a dimension of 1; Subsequently, a batch-based classification loss was employed. Combining overall and atomic similarity, the formula is: （12）； Where 𝐵 represents the batch size and 𝜏 represents the temperature coefficient. These represent the nth overall combined token and the target token in the batch, respectively; set up Represents the similarity distribution of the nth overall combined token in the batch, where The similarity between the nth overall combined token and the nth overall target token is defined as: （13）； Similarly, the similarity distribution of the nth overall target token in the batch is obtained, denoted as... And the similarity distribution of atomic combination tokens and target tokens in the batch, denoted as , respectively. Next, we define distribution regularization. as follows: （14）； Finally, the following optimization function for LUPD-NET is obtained. : （15）； Here, Θ is the LUPD-NET parameter to be learned, and 𝜅 is the trade-off hyperparameter.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the combined video retrieval method based on hierarchical uncertainty perception disambiguation as described in any one of claims 1-6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the combined video retrieval method based on hierarchical uncertainty perception disambiguation as described in any one of claims 1-6.
9. A combined video retrieval system based on hierarchical uncertainty perception disambiguation, characterized in that, include: The multimodal query combination feature generation module is configured to: understand each element of a given triple, including the reference video, the target video, and the modified text, and generate corresponding embedding features respectively; and combine and understand the reference video and the modified text to generate multimodal query combination features. The multi-level semantic alignment structure module is configured to perform cross-modal semantic alignment between multimodal query combined features and target video features to achieve combined video retrieval.
10. A combined video retrieval system based on hierarchical uncertainty perception disambiguation according to claim 9, characterized in that, The multimodal query combined feature generation module includes a global semantic reference resolution mechanism module and an atomic uncertainty modeling module; The global semantic reference resolution mechanism module is configured to: mine semantically overlapping areas through the video-text cross-modal global interaction structure, and assist in locating specific video objects from the overall semantic level by locating fuzzy references in the text; The atomic uncertainty modeling module is configured to: introduce an atomic interaction mechanism, combine Gaussian probability embedding to model uncertain semantics in the text, and refine the focus on key details in the video.