Combined video retrieval method and system based on sharing and difference semantic enhancement

Abstract

The invention relates to a combined video retrieval method and system based on sharing and difference semantic enhancement. The method comprises the following steps: inputting a reference video, a target video and a modified text into a REFINE model to realize combined video retrieval; the method specifically comprises the following steps: carrying out shared semantic enhancement, inter-frame difference decoupling and associated object aggregation on an REFINE model, and carrying out multi-modal query combination on an input reference video and a modified text to obtain a combination feature; and calculating cosine similarity of the combined features and candidate video marks in WebVid-CoVR, taking the cosine similarity as similarity scores, carrying out descending sort on the similarity scores, and selecting target videos with the similarity scores ranking the top K as required to complete combined video retrieval. According to the method, the target video of the user is effectively retrieved.

Description

Technical Field

[0001] This invention relates to a method and system for querying, understanding, and retrieving similarity data from self-media videos, and in particular to a combined video retrieval method and system based on shared and differential semantic enhancement, belonging to the field of multimodal information retrieval technology. Background Technology

[0002] Combinatorial image retrieval (CIR) is a highly challenging image retrieval task that aims to find desired images using composite queries that integrate visual input with textual modifications. In the video domain, a related retrieval paradigm, Combinatorial Video Retrieval (CVR), has been proposed. CVR allows users to retrieve desired videos by submitting multimodal queries that include a reference video and descriptive modified text. In the multimodal query, the reference video describes the user's basic retrieval needs, while the modified text expresses the user's detailed modification requirements for the reference video. Due to its outstanding ability to express complex video retrieval needs, CVR has received extensive research and attention in recent years. Furthermore, CVR research benefits various application scenarios, such as multimodal data understanding, e-commerce recommendation, and expert systems.

[0003] The key to CVR lies in accurately understanding multimodal queries and achieving effective multimodal feature fusion. It is worth noting that some pioneering efforts have focused on enhancing the model's understanding of multimodal queries. However, this invention notes that existing work has neglected to leverage the shared and variability of video to enhance multimodal query understanding. As an extension of the CIR task into the video domain, the biggest difference between the CVR task and the CIR task is the introduction of video to replace the original reference and target images. It is readily apparent in this invention that the biggest difference between video and images lies in the fact that video is a dynamic process; its frames share both commonalities (i.e., the same visual regions) and variability (i.e., different visual details).

[0004] Regarding shared features, different frames of a video may contain common objects, such as "stream" and "forest." These shared features are often strongly correlated with multimodal queries and should be enhanced in video understanding. Regarding variability, since video is a dynamic process, not all frames may independently satisfy multimodal query requirements. For example, some early frames may contain "forest" but not "stream," while later frames may contain "stream" but not "forest." Only the set of these frames can satisfy a multimodal query. Therefore, the visual detail differences between these frames also provide additional clues for multimodal query matching and should receive special attention. However, overcoming these limitations is not easy due to the following challenges:

[0005] (1) For shared features, since the present invention does not have a direct shared semantic extraction supervision signal, how to enhance the expression of shared features is the first challenge.

[0006] (2) Since the visual detail differences between frames lack direct expression, they need to be extracted from frame changes. Therefore, how to extract the differential semantics between frames is the second challenge.

[0007] (3) Since the video may contain multiple objects, the modification text may also contain modification requirements for multiple objects. Therefore, the third challenge is how to establish a connection between the modification requirements and the relevant visual objects and construct more accurate combined features. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a combined video retrieval method based on enhanced video sharing and variability, enabling effective retrieval of user-targeted videos.

[0009] The present invention also proposes a combined video retrieval system based on enhanced sharing and variability of videos.

[0010] To this end, this invention first proposes a shared feature enhancement module, which utilizes learnable labels to mine shared features between frames and achieves enhanced expression of shared features; secondly, this invention proposes a differential semantic separation module, which decouples the differential semantics between frames through a carefully designed differential block, serving as a supplement to the original video features; finally, this invention proposes an associated object aggregation module, which includes cluster alignment and aggregation, used to cluster visual and textual semantics according to modified association relationships, thereby obtaining cluster centers to guide subsequent multimodal query feature fusion.

[0011] Terminology Explanation: 1. CLIP is a deep learning model designed to combine textual and image information through contrastive learning to achieve cross-modal understanding. CLIP is pre-trained on large-scale image-text datasets and can be used for various tasks such as image retrieval, text generation, and image classification. Its ability to learn cross-modal features makes it perform well in processing visual and linguistic information.

[0012] 2. Attention mechanism is a computational method used in deep learning models, which aims to dynamically focus on specific parts of the input data to improve the efficiency and effectiveness of information processing.

[0013] 3. Multilayer Perceptron (MLP) is a feedforward neural network consisting of at least three layers: an input layer, a hidden layer, and an output layer. Each layer consists of multiple neurons connected by weights. It is widely used in tasks such as classification, regression, and feature extraction, and has shown superior performance in multiple fields such as pattern recognition, image processing, and natural language processing.

[0014] 4. The Softmax function is an activation function widely used in the output layer of multi-class classification problems. Its main function is to transform the input real-valued vector into a probability distribution, such that each output value is between 0 and 1, and the sum of all output values ​​is 1.

[0015] 5. BLIP-2 (Bootstrapping Language-Image Pretraining) is a multimodal pre-trained model for image-text tasks. It enhances the joint representation capabilities of vision and language, improving the model's performance across modalities. BLIP-2 encodes image and text information into a unified semantic space using the Q-Former architecture, supporting tasks such as image retrieval and text generation. Its innovation lies in utilizing low-cost visual feature enhancement methods, enabling the visual-language model to achieve superior performance across multiple tasks.

[0016] 6. Q-Former (Query-based Transformer) is a core component of BLIP-2 for cross-modal coding. It establishes a connection between vision and language by mapping image and text features to a unified semantic space. Q-Former encodes image features using learnable query vectors and then combines them with text features to achieve semantic alignment. It can map information from different modalities (such as images and text) to similar representations, supporting image-text matching tasks.

[0017] 7. Multi-head attention is a self-attention mechanism used in neural networks. It maps the query, key, and value of the input data to multiple subspaces, calculates attention weights in each subspace, and finally concatenates the results from each head, improving the model's ability to perceive different semantic subspaces. Multi-head attention allows the model to capture information from multiple perspectives, making it more flexible and robust.

[0018] 8. Cross-Attention is an attention mechanism commonly used in multimodal learning to connect information from different modalities. Cross-Attention treats a representation from one modality (e.g., image features) as a query and a representation from another modality (e.g., text features) as a key and value. By calculating the similarity between the query and the key, cross-Attention effectively combines visual and linguistic information to generate meaningful cross-modal representations. It is frequently used in tasks such as image-text retrieval and visual question answering.

[0019] 9. Normalization in deep learning typically refers to the operation of standardizing vectors or matrices to ensure that the data maintains an appropriate range during training. Common normalization operations include batch normalization and layer normalization. In multi-head attention, the normalization operation is used to ensure that the input and output of each layer have a stable distribution, thereby accelerating model convergence and avoiding gradient explosion or vanishing.

[0020] The technical solution of the present invention is as follows: Combined video retrieval methods based on shared and differential semantic enhancement include: The REFINE model is used to perform combined video retrieval by inputting reference videos, target videos, and modified text; specifically, this includes: The REFINE model performs multimodal query combination on the input reference video and modified text through shared semantic enhancement, inter-frame difference decoupling, and associated object aggregation to obtain combined features; Among them, shared semantic enhancement refers to: for inter-frame shared features, by introducing inter-frame shared enhancement tags, capturing the shared semantics between frames, and mining the shared nature of the reference video and the target video respectively; Inter-frame difference decoupling refers to: for the visual detail differences between frames, using differential cue attention to model the visual detail differences between frames, extracting the differential semantics between frames, thereby modeling the variability of the video, and providing additional clues for multimodal query semantic understanding; Associative object aggregation refers to: for multi-object matching and modification intent alignment, by training cluster centers, modeling the modification association of visual semantics and textual semantics, and aggregating visual and textual semantics with modification association relationships; and integrating the cluster centers expressing association information into multimodal query features through associative object aggregation to construct accurate combined features. The cosine similarity between the combined features and the candidate video tags in WebVid-CoVR is calculated as the similarity score. The similarity scores are then sorted in descending order. The top K target videos with the highest similarity scores are selected as needed to complete the combined video retrieval.

[0021] Further preferred methods include shared semantic enhancement, specifically including: The enhanced tags and frame slices are embedded and concatenated into the BLIP-2 encoder for deep interaction, capturing potential shared semantics; and the semantics of each frame are extracted. token, as a frame-level semantic representation; Using the global [CLS] representation as the query, enhanced tags as key values, multi-head attention is performed, and shared attention scores are extracted; Output shared enhanced features.

[0022] A further preferred approach involves concatenating the enhanced markers with the local visual features of each frame and inputting the result into the BLIP-2 encoder for deep interaction, capturing potential shared semantics; and extracting the semantics representing the global meaning from each frame. Tokens, serving as frame-level semantic representations; specifically including: First, define an enhancement tag for capturing inter-frame shared semantics, denoted as . ,in, Represents the number of video sampling frames. Represents the embedding dimension; for a given reference video Obtain the slice embedding of its sampled frame. ,in, This represents the number of channels in each frame; subsequently, the enhancement markers are used as a semantic enhancement supplement to the slice embedding and fed into the BLIP-2 encoder for visual semantic representation, as shown in the following formula: ; in, Represents the BLIP-2 encoder. Features of the reference video sampling frames; Subsequently, the global semantics of each frame were extracted. tokens, forming frame-level semantic representations .

[0023] Further preferred, frame global features are used as , will use the enhanced tag as and Multi-head attention interaction is performed, and inter-frame shared semantic aggregation is executed, as follows: ; in, The shared features representing the reference video are called shared enhancement features; It is a multi-head attention mechanism; For the target video, obtain the shared features of the target video. .

[0024] A further preferred method is inter-frame difference decoupling, which specifically includes: Using the obtained shared features, the differential representation between adjacent frames is calculated as a latent differential semantics; a differential cue attention mechanism is introduced to perform attention interaction between the inter-frame differential semantics and the original shared features. By summing the difference results of all adjacent frames, the variability features of the reference video are obtained; and an interpolation fusion strategy is introduced to stitch together shared features and difference features alternately to construct a multi-level, multi-source fine-grained semantic representation.

[0025] Further preferred, for the reference video, the first Frame sharing features First, calculate Adjacent to it The semantic difference of the frames, the result is This is considered as the global difference semantics between the two; differential cue attention is introduced, which includes a single-layer transformer and a sigmoid activation function. By interacting with the global difference semantics between frames and the original frame semantics through attention, it effectively captures visual differences in details, as formulated as follows: ; in, Representing the Frame and the The difference features between frames, DCA is an abbreviation for Differential Cueing Attention.

[0026] Further preferred, the differential features between all frames and their adjacent frames are obtained to form the variability features of the reference video. The formula is as follows: ; Will Interpolation connections are performed to achieve smooth inter-frame sharing and interaction of variable information, as formulated below. ; in, Interpolation features representing the reference video; For the target video, obtain the interpolation features of the target video. .

[0027] Further preferred, for multi-object matching and modification intent alignment, by training cluster centers, the modification association between visual semantics and textual semantics is modeled, and visual and textual semantics with modification associations are aggregated; specifically including: By using a pre-trained Q-Former, features from the reference video, target video, and modified text are mapped to a unified semantic space; By introducing a shared clustering center and assigning alignment weights through interaction with different modal tags, visual and textual components with modified semantic associations are aggregated. Related semantic fragments in different modalities are guided to cluster together to the same cluster center, achieving semantic alignment between the modification intention and the visual object.

[0028] Further preferred, by utilizing pre-trained Q-Former, features from the reference video, target video, and modified text are mapped to a unified semantic space; Using a Q-Former pre-trained from BLIP-2, the features of each element in the triples are mapped to a unified semantic space, formalized as follows: ; in, This indicates that the Q-Former defined in BLIP-2 is learnable and queryable. Indicate its number; and These represent the interpolation features of the obtained reference video and target video, respectively; , , These represent the reference marker, target marker, and modification marker, respectively, which are mapped to the unified semantic space via Q-Former; This indicates that the text has been modified. This refers to the BLIP-2 text encoder. Q-Former is the core component in BLIP-2 used for cross-modal coding.

[0029] Further optimization involves introducing a modality-sharing clustering center. ,in This represents the total number of cluster centers. Interactions occur between each modality-shared cluster center and all elements of the triple; the alignment weights of the modality-shared cluster centers are assigned through interactions between different modalities; for the reference video, the alignment weights... The definition is as follows: ; Where s represents the cosine similarity function, The first feature representing the reference video q One tag, It is the nth cluster center, and 𝜏 is the temperature parameter; Alignment weights are then assigned to the corresponding cluster centers. The formula is as follows: ; Norm is a normalization operation.

[0030] A further preferred embodiment integrates cluster centers expressing correlation information into multimodal query features via an aggregation module to construct accurate combined features; specifically including: Each tag is matched with the most relevant cluster center, and its associated semantics are aggregated into the features through cross attention; this step is performed on the reference video, the target video, and the modified text to obtain the enhanced triplet features; The MLP is used to learn the modification associations between reference videos and modified text, and weighted fusion is used to generate combined query features; By comparing the loss optimization features with the semantic consistency between them and the target video, retrieval accuracy is improved.

[0031] Further optimization involves, for the reference video, first identifying the cluster center that most closely resembles the reference marker. The formula is as follows: ; Subsequently, Cross-attention interaction is performed with the q-th marker in the reference video to aggregate the relationships to the corresponding markers, as formulated as follows: ; in, It is the first The enhanced features of the reference video tags, Cross-Att refers to the cross-attention mechanism; Obtain the associated features of the reference video ; For the target video and the modified text, the association features between the target video and the modified text are obtained, and are denoted as follows: , .

[0032] A further preferred approach involves using MLP to learn the modification associations between reference videos and modified text, and then weighted and fused to generate combined query features; including: First, we use MLP to learn the modification association between each tag and the modified text in the reference video, in the following form. ; in, ; Subsequently, the modified associations are aggregated into the reference video and the modified text respectively, and then combined by addition to form the association semantics, as follows: ; in, This represents the combined query characteristics.

[0033] A further preferred approach is to use batch-based classification loss. The combined query features are then pushed closer to the target video features, as formulated as follows: ; Where B is the size of the batch. This represents the combined feature of the i-th average pooling operation in the batch. This represents the target video feature after average pooling in the i-th batch. For temperature coefficient, This represents the cosine similarity function.

[0034] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described combined video retrieval method based on shared and differential semantic enhancement.

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described combined video retrieval method based on shared and differential semantic enhancement.

[0036] A video retrieval system based on shared and differential semantic enhancement includes: The shared feature enhancement module is configured to: for inter-frame shared features, effectively capture the shared semantics between frames by introducing enhanced inter-frame shared tags, mine the shared nature of the reference video and the target video respectively, and enhance the model's ability to perceive and represent the visual shared semantics in triples; The differential semantic separation module is configured to: model the visual detail differences between frames with differential cue attention for the visual detail differences between frames, extract the differential semantics between frames, thereby modeling the variability of the video and providing additional cues for multimodal query semantic understanding; The associated object aggregation module is configured to: for multi-object matching and modification intent alignment, model the modification association of visual semantics and textual semantics by training cluster centers, and aggregate visual and textual semantics with modification association relationships; and integrate the cluster centers expressing association information into multimodal query features through the aggregation module to construct accurate combined features. The video retrieval module is configured to use multimodal combined features to filter videos in the candidate video set and obtain the formal result set, i.e., the final result of combined video retrieval.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a shared feature enhancement module, which helps to mine stable shared semantics between video frames. By introducing learnable shared tags, it improves the model's ability to model visual commonalities in reference and target videos, thereby enhancing the consistency of shared semantic representation in triples.

[0038] 2. This invention proposes a differential semantic separation module, which can extract key change clues based on inter-frame differences and accurately capture fine-grained visual changes with the help of differential attention mechanism, effectively improving the model's ability to perceive dynamic changes in videos and providing structured change information support for multimodal queries.

[0039] 3. This invention proposes an associated object aggregation module, which achieves semantic alignment between text modification intent and video visual objects by training modality sharing cluster centers, and then injects associated information into triple features to improve the semantic integrity and discriminative ability of combined features. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the combined video retrieval method based on shared and differential semantic enhancement of the present invention. Detailed Implementation

[0041] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0042] Example 1 A combined video retrieval method based on shared and differential semantic enhancement (REFINE), such as Figure 1 As shown, it includes: The REFINE model is used to perform combined video retrieval by inputting reference videos, target videos, and modified text; specifically, this includes: The REFINE model performs multimodal query combination on the input reference video and modified text through shared semantic enhancement, inter-frame difference decoupling, and associated object aggregation to obtain combined features; Among them, shared semantic enhancement refers to: for inter-frame shared features, by introducing inter-frame shared enhancement tags, effectively capturing the shared semantics between frames, and mining the sharing between the reference video and the target video respectively; enhancing the model's ability to perceive and represent visual shared semantics in triples; Inter-frame difference decoupling refers to: for the visual detail differences between frames, using differential cue attention to model the visual detail differences between frames, extracting the differential semantics between frames, thereby modeling the variability of the video, and providing additional clues for multimodal query semantic understanding; Associative object aggregation refers to: for multi-object matching and modification intent alignment, by training cluster centers, modeling the modification association of visual semantics and textual semantics, and aggregating visual and textual semantics with modification association relationships; and integrating the cluster centers expressing association information into multimodal query features through associative object aggregation to construct accurate combined features. The cosine similarity between the combined features and candidate video tags in WebVid-CoVR is calculated as the similarity score. These scores are then sorted in descending order. Based on the needs (e.g., retrieving the K videos that best meet the query requirements), the top K target videos by similarity score are selected (this forms the final result set), completing the combined video retrieval. WebVid-CoVR is a large-scale benchmark dataset designed specifically for CVR tasks. This dataset contains 1.6 million CVR triples, covering approximately 131,000 different videos and 467,000 unique modified texts. On average, each video is approximately 16.8 seconds long, and each modified text contains approximately 4.8 words. Furthermore, each target video is associated with an average of approximately 12.7 triples. The WebVid-CoVR test subset consists of 2,500 high-quality test triples, rigorously labeled and cleaned of noise, providing a robust and challenging evaluation benchmark for the model.

[0043] Example 2 The difference between the combined video retrieval method based on shared and differential semantic enhancement described in Example 1 and the following is: Shared semantic enhancement, specifically including: The enhanced tags and frame slices are embedded and concatenated into the BLIP-2 encoder for deep interaction, capturing potential shared semantics; and the semantics of each frame are extracted. token, as a frame-level semantic representation; Using the global [CLS] representation as the query, enhanced tags as key values, multi-head attention is performed, and shared attention scores are extracted; Output shared enhanced features. Used for subsequent difference modeling and combined semantic understanding.

[0044] The enhanced tags are concatenated with the local visual features of each frame and input into the BLIP-2 encoder for deep interaction to capture potential shared semantics; and the global semantics of each frame are extracted. Tokens, serving as frame-level semantic representations; specifically including: To enhance the understanding of key visual semantics by leveraging the shared nature of video, this method designs a shared semantic enhancement approach. By introducing enhancement markers for shared parameters between frames during the BLIP-2 visual coding process, it efficiently captures shared semantics between frames, enhancing the visual encoder's ability to encode shared semantic content between frames. This achieves shared semantic enhancement for both the reference video and the target video. The following section uses the reference video as an example. For example, let's elaborate on this part:

[0045] First, define an enhancement tag for capturing inter-frame shared semantics, denoted as . ,in, Represents the number of video sampling frames. This represents the embedding dimension; to enhance the awareness of shared semantics by the enhanced token, the module uses it as part of the input to the BLIP-2 image encoder, allowing for full interaction with the original token in ViT. Specifically, for a given reference video... Obtain the slice embedding of its sampled frame. ,in, Represents the number of channels in each frame ( The number of slices per frame increases by one. (Tags); Subsequently, the enhanced tags are used as semantic enhancement supplements to the slice embeddings and fed into the BLIP-2 encoder for visual semantic representation, as shown in the following formula:

[0046] ; in, Represents the BLIP-2 encoder. Features of the reference video sampling frames; Subsequently, the global semantics of each frame were extracted. tokens, forming frame-level semantic representations .

[0047] To further analyze the inter-frame shared semantics captured in the augmented tags and effectively integrate these shared semantics into the semantic embeddings of each frame, frame global features are used as... , will use the enhanced tag as and Multi-head attention interaction is performed, and inter-frame shared semantic aggregation is executed, as follows: ; in, The shared features representing the reference video are called shared enhancement features; It is a multi-head attention mechanism; Similarly, for the target video, this method obtains the shared features of the target video through the same shared semantic enhancement process. .

[0048] Inter-frame difference decoupling: For visual detail differences between frames, differential cue attention is used to model the visual detail differences between frames, extracting the semantic differences between frames to model the variability of the video, providing additional clues for multimodal query semantic understanding, specifically including: Using the obtained shared features, the differential representation between adjacent frames is calculated as a latent differential semantics; a differential cue attention mechanism is introduced to perform attention interaction between the inter-frame differential semantics and the original shared features. By summing the difference results of all adjacent frames, the variability features of the reference video are obtained; and an interpolation fusion strategy is introduced to stitch together shared features and difference features alternately to construct a multi-level, multi-source fine-grained semantic representation.

[0049] By leveraging the shared features of the previously obtained reference and target videos, and in order to utilize the variability enhancement model for video understanding, this method designs differential semantic separation. This involves decoupling the latent differential semantics between frames through differential blocks to model the visual detail differences between frames. The implementation details of the differential blocks are described below:

[0050] To fully exploit potential differential semantics, differential blocks are designed to extract differential semantics from all adjacent frame pairs in the video sampling frames. Taking a sample frame of the reference video as an example, specifically, for the first sample frame of the reference video... Frame sharing features First, calculate Adjacent to it The semantic difference of the frames, the result is This is considered as the global difference semantics between the two frames. However, if this difference semantics is directly used as the final difference semantic expression, the lack of sufficient interaction with the original frame may cause the subtle differences between two adjacent frames to be ignored. To achieve more accurate decoupling of the difference semantics between frames, differential cue attention is introduced. Differential cue attention includes a single-layer transformer and a sigmoid activation function. By interacting with the global difference semantics between frames and the semantics of the original frame through attention, it effectively captures subtle visual differences, as formulated as follows:

[0051] ; in, Representing the Frame and the The difference features between frames, DCA is an abbreviation for Differential Cueing Attention.

[0052] In the same manner, the differential features between all frames and their adjacent frames are obtained to form the variability features of the reference video. The formula is as follows: ; This method aims to interpolate and fuse shared semantics and differential features between video frames to construct a multi-level, multi-source feature representation, enhancing the fine-grainedness and diversity of video semantic expression and providing rich structural support for query combination. Finally, to effectively fuse shared and variable features between video frames in subsequent associated object aggregation, the method will... Interpolation connections are performed to achieve smooth inter-frame sharing and interaction of variable information, as formulated below.

[0053] ; in, Interpolation features representing the reference video; Similarly, for the target video, this method also uses difference blocks to mine its difference semantics and obtain the interpolation features of the target video. .

[0054] Therefore, the multi-level, multi-source fine-grained semantic representation is formed by alternately concatenating frame-level shared features from the shared semantic enhancement module and inter-frame differential features from the inter-frame difference decoupling module in this method. , This step not only integrates different types of temporal semantics but also enhances the model's ability to express the semantic structure within the video, making it a key step in supporting the subsequent aggregation of related objects.

[0055] For multi-object matching and modification intent alignment, by training cluster centers, the modification association between visual and textual semantics is modeled, and visual and textual semantics with modification associations are aggregated; specifically including: By using pre-trained Q-Former, features from reference video, target video, and modified text are mapped to a unified semantic space; a foundation for cross-modal alignment is established. By introducing a shared clustering center and assigning alignment weights through interaction with different modal tags, visual and textual components with modified semantic associations are aggregated. Related semantic fragments from different modalities are guided to cluster together at the same cluster center, achieving semantic alignment between the modification intent and the visual object. This provides support for subsequent combined feature modeling.

[0056] By utilizing a pre-trained Q-Former, features from the reference video, target video, and modified text are mapped to a unified semantic space. This method aims to construct a unified semantic space and introduce modality-shared clustering centers. Guided clustering is used to align and model semantic segments in the video and text, thereby clarifying the semantic relationship between the modification intent in the text and the visual objects in the video, and improving the mapping accuracy between multimodal objects.

[0057] This section aims to model modification associations in multimodal queries, thereby guiding subsequent feature combinations for multimodal queries. Specifically, to uncover modification associations, a Q-Former pre-trained from BLIP-2 is used to map the features of each element in the triples to a unified semantic space, formalized as follows:

[0058] ; in, This indicates that the Q-Former defined in BLIP-2 is learnable and queryable. Indicate its number; and These represent the interpolation features of the obtained reference video and target video, respectively; , , These represent the reference marker, target marker, and modification marker, respectively, which are mapped to the unified semantic space via Q-Former; This indicates that the text has been modified. This refers to the BLIP-2 text encoder. Q-Former is the core component in BLIP-2 used for cross-modal coding.

[0059] To model modification associations in multimodal queries, this module introduces modality-shared clustering centers. ,in This represents the total number of cluster centers. It is worth noting that during training, to ensure that each cluster center accurately aggregates labels with modified associations, interactions are performed between each modality-shared cluster center and all elements of the triples (including the target video); the alignment weights of the modality-shared cluster centers are assigned through interactions between different modalities; for the reference video, the alignment weights... The definition is as follows: ; Where s represents the cosine similarity function, The first feature representing the reference video q One tag, It is the nth cluster center, and 𝜏 is the temperature parameter; Alignment weights are then assigned to the corresponding cluster centers. The formula is as follows: ; Norm is a normalization operation that maintains a stable numerical range.

[0060] Similarly, target video tagging and modify text tags It also interacts with the cluster center in the same way. Through this process, each modification intent is associated with a shared cluster, where the corresponding text and video tags are aggregated. The corresponding modification associations are then passed through the cluster center.

[0061] Cluster centers expressing correlation information are integrated into multimodal query features through an aggregation module to construct accurate combined features; specifically including: Each tag is matched with the most relevant cluster center, and its associated semantics are aggregated into the features through cross attention; this step is performed on the reference video, the target video, and the modified text to obtain the enhanced triplet features; The MLP is used to learn the modification associations between reference videos and modified text, and weighted fusion is used to generate combined query features; By comparing the loss optimization features with the semantic consistency between them and the target video, retrieval accuracy is improved.

[0062] Guided by cluster centers, this method aims to efficiently aggregate relevant semantic components from video and text modalities, constructing a context-aware and semantically consistent fusion feature representation. This mechanism strengthens the dominant role of modifying relevant semantics in the final combined representation, effectively combating semantic drift.

[0063] After obtaining cluster centers containing modified associations, this method designs an association semantic aggregation, aggregating the learned associations into the features of each element in the triples. This further enhances the feature representation of the triples, facilitating the construction of more accurate combined features.

[0064] For the reference video, first identify the cluster center that is most similar to the reference marker. The formula is as follows: ; Subsequently, Cross-attention interaction is performed with the q-th marker in the reference video to aggregate the relationships to the corresponding markers, as formulated as follows: ; in, It is the first The enhanced features of the reference video tags, Cross-Att refers to the cross-attention mechanism; For the remaining markers in the reference video, they are all aggregated with their corresponding cluster centers using the method described above to obtain the reference video association features. ; Similarly, the same aggregation operation is performed on the target video and the modified text to obtain the association features between the target video and the modified text, which are denoted as follows: , .

[0065] The MLP is used to learn the modification associations between reference videos and modified text, and weighted fusion is used to generate combined query features; including: In this method, after completing the triplet feature representation, the multimodal queries will be combined. Specifically, based on previous research, firstly, an MLP is used to learn the modification association between each tag and the modified text in the reference video, in the following form:

[0066] ; in, ; Subsequently, the modified associations are aggregated into the reference video and the modified text respectively, and then combined by addition to form the association semantics, as follows: ; in, This represents the combined query characteristics.

[0067] Using batch-based classification loss The combined query features are then pushed closer to the target video features, as formulated as follows: ; Where B is the size of the batch. This represents the combined feature of the i-th average pooling operation in the batch. This represents the target video feature after average pooling in the i-th batch. For temperature coefficient, This represents the cosine similarity function.

[0068] Table 1 is a schematic diagram showing the comparison of the retrieval accuracy of the present invention on the WebVid-CoVR dataset; Table 1

[0069] Table 2 is a schematic diagram showing the comparison of the retrieval accuracy of the present invention on the FashionIQ dataset; Table 2

[0070] Table 3 is a schematic diagram showing the comparison of the retrieval accuracy of the present invention on the Shoes dataset; Table 3

[0071] Table 4 is a schematic diagram showing the comparison of the retrieval accuracy of the present invention on the CIRR dataset; Table 4

[0072] For CVR tasks, this embodiment uses the following baseline: Text-Only: Modified text is used only during training and inference, without being fused with the reference video. This setting isolates performance contributed solely by text input. This invention uses BLIP-2 as the backbone network.

[0073] Video-Only: Training and inference are performed using only reference videos. This method typically measures the visual similarity between the reference and target videos without relying on any modified text. This invention selects BLIP-2 as the backbone network.

[0074] CoVR, presented at the AAAI conference in 2024, is one of the pioneering works in the CVR task. It learns combined cross-modal features by jointly encoding reference videos and modified text through a cross-attention mechanism. The model is trained on the large-scale WebVid-CoVR dataset and fine-tuned on the text encoder to better support the performance of CVR and CIR. In the settings of this invention, the experimental protocol is followed, and results are reported under different backbone networks, including CLIP, BLIP, and BLIP-2.

[0075] CoVRSEE, presented at CVPR 2023, leverages detailed linguistic descriptions to effectively encode contextual information relevant to specific queries and learns powerful embedding representations, thereby enhancing alignment and aiding in the retrieval of the most relevant target videos.

[0076] FDCA, published at the ICLR conference in 2025, proposed a method for text feature decoupling and cross-modal alignment. It decouples and modifies text at the sentence and tag levels, performs tag-level feature clustering, and decouples the tags into three types, ultimately generating semantically rich video features, thereby improving retrieval accuracy.

[0077] For CIR tasks, this embodiment uses the following baseline: TIRG, presented at the 2019 CVPR conference, proposed an innovative gated residual connection designed for combinatorial image retrieval (CIR). It preserves the original image features while integrating additional features by combining image and text data, thereby improving the ability of feature representation.

[0078] VAL, presented at the CVPR conference in 2020, proposed a unified CIR framework that integrates multi-granular visual-linguistic features to jointly preserve and transform visual content, and processes it based on text instructions.

[0079] CLVC-Net, published at the SIGIR conference in 2021, proposed a comprehensive CIR composition framework that explicitly models fine-grained text-image alignment at both the local and global levels through independent composition modules and affine transformations.

[0080] CIRPLANT, presented at the ICCV conference in 2021, proposes CIR using large-scale vision and language pre-trained (VLP) transformers, effectively extending the traditional metric learning pipeline to handle real-world images in open domains.

[0081] ARTEMIS, presented at the ICLR conference in 2022, introduced a two-module framework for image retrieval with free-form text modifiers. It combines cross-modal and visual retrieval by jointly modeling explicit matching of attributes of text descriptions and implicit similarity with reference images.

[0082] FashionVLP, presented at the CVPR conference in 2022, introduced a vision-language pre-trained (VLP) model for fashion retrieval from text input, leveraging multi-layered fashion-specific visual context and large-scale vision-language pre-training.

[0083] SSN, presented at the AAAI conference in 2024, reconstructed CIR by using a text input framework as instructions to describe semantic changes from a reference image to a target image, and explicitly decomposed it into degradation and upgrade components.

[0084] SADN, presented at the ACM MM conference in 2024, addresses the issues of false negatives and subtle semantic differences in CIR by leveraging neighborhood-based semantic relevance and discriminative modeling.

[0085] ComqueryFormer, published in the TMM journal in 2024, proposed a unified Transformer-based CIR framework that effectively mitigates encoder discrepancies by uniformly encoding multimodal inputs.

[0086] LF-CLIP, presented at the CVPR conference in 2022, uses CLIP to encode visual and textual information separately and fuses them through a simple network, demonstrating superior performance in combined image retrieval tasks.

[0087] CRN, published in the TIP journal in 2023, introduced a cross-relation network for CIR, which significantly reduced computational complexity by dynamically balancing retrieval strategies between modified and auxiliary query texts through relevance modeling and a shared feature extractor.

[0088] MGUR, presented at the ICLR conference in 2024, proposed a retrieval method based on multi-granularity uncertainty, which combines fine-grained one-to-one matching with coarse-grained one-to-many matching to effectively handle fuzzy text queries and imprecise visual representations.

[0089] Published in the TOIS journal in 2025, IUDC proposed an enhanced dual-channel matching framework based on a large-scale language model (LLM) that effectively performs semantic reasoning through multi-level attribute extraction and performs fine-grained matching of multimodal embeddings.

[0090] Presented at the ACM MM conference in 2023, TG-CIR models the conflict relationship between the goal and the query through a "keep and replace" strategy and captures complex user intent by leveraging teacher-student combined branches, achieving strong performance in multiple benchmarks.

[0091] CompoDiff, in his 2023 TMLR paper, treats CIR as a conditional image editing problem in a latent space and designs a diffusion-based CIR model to solve the task.

[0092] BLIP4CIR, presented at the CVPR conference in 2024, proposes a bidirectional training method that utilizes a pre-trained visual-language text encoder and effectively captures reverse query semantics through learnable tokens.

[0093] CASE, presented at the AAAI conference in 2024, proposes a cross-attention driven CIR offset encoder that leverages the bimodal cross-attention of BLIP to dynamically adjust the query-image embedding toward the target representation.

[0094] Candidate, published in the TMLR journal in 2024, proposed a two-stage CIR pipeline that efficiently filters out easy negative samples based on visual similarity and explicitly reorders the remaining difficult negative samples using fine-grained text-image interaction modeling.

[0095] SPRC, presented at the ICLR conference in 2024, proposed a new CIR method based on sentence-level prompts. It utilizes a lightweight query Transformer (Q-Former) to generate expressive prompts by jointly modeling reference images and relative descriptions.

[0096] CoVR-2, published in the TPAMI journal in 2024, has also been evaluated for CIR performance on fashion and open-domain datasets, and its performance in CVR tasks also shows great potential.

[0097] Recall refers to the proportion of queries whose relevant items are successfully retrieved out of the top K results. Average Recall (AVR) is an average of several Recall@K metrics, typically used to comprehensively evaluate a model's performance at different K values. It is calculated by averaging the recall rates for different K values ​​(e.g., K = 1, 5, 10).

[0098] On the WebVid-CoVR dataset, the Recall metric measures the degree of matching between the model and the modified text, while the average recall is the average of all Recall metrics. Common Recall metrics include Recall@1, Recall@5, and Recall@10, which represent the proportion of the correct target found in the first 1, 5, and 10 results, respectively.

[0099] On FashionIQ, the Recall metric is used to evaluate the model's ability to match clothing images with their corresponding text descriptions. For example, Recall@10 and Recall@50 indicate whether the target image is present in the first 10 and first 50 images returned by the model, respectively. Average recall, on the other hand, is calculated by taking the average of performance metrics (such as Recall@K) for different clothing categories to comprehensively evaluate the model's performance across all categories.

[0100] The Recall metric on the Shoes dataset is used to evaluate the matching effect between the retrieved shoe images and text descriptions. Its average recall is calculated by averaging all Recall metrics. Similar to other datasets, Recall@K measures whether a target image relevant to the query text can be found among the top K returned results.

[0101] Tables 1-4 show the comparison and results of query efficiency and retrieval accuracy with leading international methods of the same type. The results demonstrate that, compared to similar combined video retrieval methods, the RIFINE model of this invention achieves higher accuracy on widely used benchmark datasets.

[0102] Example 3 A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the combined video retrieval method based on shared and differential semantic enhancement as described in Embodiment 1 or 2.

[0103] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the combined video retrieval method based on shared and differential semantic enhancement as described in Embodiment 1 or 2.

[0104] Example 5 A video retrieval system based on shared and differential semantic enhancement includes: The shared feature enhancement module is configured to: for inter-frame shared features, effectively capture the shared semantics between frames by introducing inter-frame shared enhancement tokens, mine the shared nature of the reference video and the target video respectively, and enhance the model's ability to perceive and represent the visual shared semantics in the triples; its core design includes introducing shared enhancement tokens, performing feature concatenation and deep interaction, and finally generating the shared semantic features of the reference video and the target video through cross-frame shared modeling.

[0105] The differential semantic separation module is configured to: model the visual detail differences between frames using differential cue attention, extract the differential semantics between frames, and thus model the variability of the video, providing additional clues for multimodal query semantic understanding; the goal of the differential semantic separation module is to parse the inter-frame differences and model video variations. Its core design includes modeling full-frame differences, introducing differential attention (DCA), and adopting an interpolation fusion strategy: alternately concatenating shared features and differential features to obtain the final frame sequence representation, enhancing the multi-granularity video semantic modeling capability.

[0106] The associated object aggregation module is configured to: for multi-object matching and modification intent alignment, model the modification association between visual and textual semantics by training cluster centers, and aggregate visual and textual semantics with modification association relationships; integrate the cluster centers expressing association information into multimodal query features through the aggregation module to construct accurate combined features; the goal of the associated object aggregation module is to model the modification association between visual and linguistic features and complete the combination of multimodal features. Its core design includes multimodal shared cluster centers, multimodal alignment using Q-Former mapping, aggregation of associated semantics, and finally, multimodal combined features to achieve matching learning.

[0107] The video retrieval module is configured to use multimodal combined features to filter videos in the candidate video set and obtain the formal result set, i.e., the final result of combined video retrieval.

Claims

1. A combined video retrieval method based on shared and differential semantic enhancement, characterized in that, include: The REFINE model is used to perform combined video retrieval by inputting reference videos, target videos, and modified text; specifically, this includes: The REFINE model performs multimodal query combination on the input reference video and modified text through shared semantic enhancement, inter-frame difference decoupling, and associated object aggregation to obtain combined features; Among them, shared semantic enhancement refers to: for inter-frame shared features, by introducing inter-frame shared enhancement tags, capturing the shared semantics between frames, and mining the shared nature of the reference video and the target video respectively; Inter-frame difference decoupling refers to: for the visual detail differences between frames, using differential cue attention to model the visual detail differences between frames, extracting the differential semantics between frames, thereby modeling the variability of the video, and providing additional clues for multimodal query semantic understanding; Associative object aggregation refers to: for multi-object matching and modification intent alignment, by training cluster centers, modeling the modification association of visual semantics and textual semantics, and aggregating visual and textual semantics with modification association relationships; and integrating the cluster centers expressing association information into multimodal query features through associative object aggregation to construct accurate combined features. The cosine similarity between the combined features and the candidate video tags in WebVid-CoVR is calculated as the similarity score. The similarity scores are then sorted in descending order. The top K target videos with the highest similarity scores are selected as needed to complete the combined video retrieval.

2. The combined video retrieval method based on shared and differential semantic enhancement according to claim 1, characterized in that, Shared semantic enhancement, specifically including: The enhanced tags and frame slices are embedded and concatenated into the BLIP-2 encoder for deep interaction, capturing potential shared semantics; and the semantics of each frame are extracted. token, as a frame-level semantic representation; Using the global [CLS] representation as the query, enhanced tags as key values, multi-head attention is performed, and shared attention scores are extracted; Output shared enhanced features; A further preferred approach involves concatenating the enhanced markers with the local visual features of each frame and inputting the result into the BLIP-2 encoder for deep interaction, capturing potential shared semantics; and extracting the semantics representing the global meaning from each frame. Tokens, serving as frame-level semantic representations; specifically including: First, define an enhancement tag for capturing inter-frame shared semantics, denoted as . ,in, Represents the number of video sampling frames. Represents the embedding dimension; for a given reference video Obtain the slice embedding of its sampled frame. ,in, This represents the number of channels in each frame; subsequently, the enhancement markers are used as a semantic enhancement supplement to the slice embedding and fed into the BLIP-2 encoder for visual semantic representation, as shown in the following formula: ； in, Represents the BLIP-2 encoder. Features of the reference video sampling frames; Subsequently, the global semantics of each frame were extracted. tokens, forming frame-level semantic representations ; Further preferred, frame global features are used as , will use the enhanced tag as and Multi-head attention interaction is performed, and inter-frame shared semantic aggregation is executed, as follows: ； in, The shared features representing the reference video are called shared enhancement features; It is a multi-head attention mechanism; For the target video, obtain the shared features of the target video. .

3. The combined video retrieval method based on shared and differential semantic enhancement according to claim 1, characterized in that, Inter-frame difference decoupling specifically includes: Using the obtained shared features, the differential representation between adjacent frames is calculated as a latent differential semantics; a differential cue attention mechanism is introduced to perform attention interaction between the inter-frame differential semantics and the original shared features. By summing the difference results of all adjacent frames, the variability features of the reference video are obtained; and an interpolation fusion strategy is introduced to stitch together shared features and difference features alternately to construct a multi-level, multi-source fine-grained semantic representation. Further preferred, for the reference video, the first Frame sharing features First, calculate Adjacent to it The semantic difference of the frames, the result is This is considered as the global difference semantics between the two; differential cue attention is introduced, which includes a single-layer transformer and a sigmoid activation function. By interacting with the global difference semantics between frames and the original frame semantics through attention, it effectively captures visual differences in details, as formulated as follows: ； in, Representing the Frame and the The difference features between frames, DCA is an abbreviation for Differential Cueing Attention; Further preferred, the differential features between all frames and their adjacent frames are obtained to form the variability features of the reference video. The formula is as follows: ； Will Interpolation connections are performed to achieve smooth inter-frame sharing and interaction of variable information, as formulated below. ； in, Interpolation features representing the reference video; For the target video, obtain the interpolation features of the target video. .

4. The combined video retrieval method based on shared and differential semantic enhancement according to claim 1, characterized in that, For multi-object matching and modification intent alignment, by training cluster centers, the modification association between visual and textual semantics is modeled, and visual and textual semantics with modification associations are aggregated; specifically including: By using a pre-trained Q-Former, features from the reference video, target video, and modified text are mapped to a unified semantic space; By introducing a shared clustering center and assigning alignment weights through interaction with different modal tags, visual and textual components with modified semantic associations are aggregated. Related semantic fragments in different modalities are guided to cluster together to the same cluster center, achieving semantic alignment between modification intentions and visual objects; Further preferred, by utilizing pre-trained Q-Former, features from the reference video, target video, and modified text are mapped to a unified semantic space; Using a Q-Former pre-trained from BLIP-2, the features of each element in the triples are mapped to a unified semantic space, formalized as follows: ； in, This indicates that the Q-Former defined in BLIP-2 is learnable and queryable. Indicate its number; and These represent the interpolation features of the obtained reference video and target video, respectively; , , These represent the reference marker, target marker, and modification marker, respectively, which are mapped to the unified semantic space via Q-Former; This indicates that the text has been modified. This refers to the BLIP-2 text encoder. Q-Former is the core component in BLIP-2 used for cross-modal coding. Further optimization involves introducing a modality-sharing clustering center. ,in This represents the total number of cluster centers. Interactions occur between each modality-shared cluster center and all elements of the triple; the alignment weights of the modality-shared cluster centers are assigned through interactions between different modalities; for the reference video, the alignment weights... The definition is as follows: ； Where s represents the cosine similarity function, The first feature representing the reference video q One tag, It is the nth cluster center, and 𝜏 is the temperature parameter; Alignment weights are then assigned to the corresponding cluster centers. The formula is as follows: ； Norm is a normalization operation.

5. The combined video retrieval method based on shared and differential semantic enhancement according to claim 1, characterized in that, Cluster centers expressing correlation information are integrated into multimodal query features through an aggregation module to construct accurate combined features; specifically including: Each tag is matched with the most relevant cluster center, and its associated semantics are aggregated into the features through cross attention; this step is performed on the reference video, the target video, and the modified text to obtain the enhanced triplet features; The MLP is used to learn the modification associations between reference videos and modified text, and weighted fusion is used to generate combined query features; By comparing the loss optimization features with the semantic consistency between them and the target video, retrieval accuracy is improved.

6. The combined video retrieval method based on shared and differential semantic enhancement according to claim 5, characterized in that, For the reference video, first identify the cluster center that is most similar to the reference marker. The formula is as follows: ； Subsequently, Cross-attention interaction is performed with the q-th marker in the reference video to aggregate the relationships to the corresponding markers, as formulated as follows: ； in, It is the first The enhanced features of the reference video tags, Cross-Att refers to the cross-attention mechanism; Obtain the associated features of the reference video ; For the target video and the modified text, the association features between the target video and the modified text are obtained, and are denoted as follows: , ; A further preferred approach involves using MLP to learn the modification associations between reference videos and modified text, and then weighted and fused to generate combined query features; including: First, we use MLP to learn the modification association between each tag and the modified text in the reference video, in the following form. ； in, ; Subsequently, the modified associations are aggregated into the reference video and the modified text respectively, and then combined by addition to form the association semantics, as follows: ； in, This represents the combined query characteristics.

7. The combined video retrieval method based on shared and differential semantic enhancement according to claims 1-6, characterized in that, Using batch-based classification loss The combined query features are then pushed closer to the target video features, as formulated as follows: ； Where B is the size of the batch. This represents the combined feature of the i-th average pooling operation in the batch. This represents the target video feature after average pooling in the i-th batch. For temperature coefficient, This represents the cosine similarity function.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the combined video retrieval method based on shared and differential semantic enhancement as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the combined video retrieval method based on shared and differential semantic enhancement as described in any one of claims 1-7.

10. A video retrieval system based on a combination of shared and differential semantic enhancement, characterized in that: include: The shared feature enhancement module is configured to: for inter-frame shared features, effectively capture the shared semantics between frames by introducing enhanced inter-frame shared tags, mine the shared nature of the reference video and the target video respectively, and enhance the model's ability to perceive and represent the visual shared semantics in triples; The differential semantic separation module is configured to: model the visual detail differences between frames with differential cue attention for the visual detail differences between frames, extract the differential semantics between frames, thereby modeling the variability of the video and providing additional cues for multimodal query semantic understanding; The associated object aggregation module is configured to: for multi-object matching and modification intent alignment, model the modification association of visual semantics and textual semantics by training cluster centers, and aggregate visual and textual semantics with modification association relationships; and integrate the cluster centers expressing association information into multimodal query features through the aggregation module to construct accurate combined features. The video retrieval module is configured to use multimodal combined features to filter videos in the candidate video set and obtain the formal result set, i.e., the final result of combined video retrieval.

Images