Fine-grained-driven video time retrieval and highlight detection joint task decoupling method

Abstract

The invention discloses a fine-grained-driven video time retrieval and highlight detection joint task decoupling method, and belongs to the technical field of computer vision. Aiming at the problems of lack of fine-grained modeling, rough cross-modal alignment and lack of task-specific modeling of an existing model, a multi-granularity video feature modeling mode is adopted, a joint task is driven through semantic supplementation among three different force features, local entity relationships and space-time details are captured, and a multi-granularity video feature model is established. And the fusion of the multi-granularity video features is realized by using the designed gating adaptive fusion module. By adopting multi-granularity feature alignment loss, accurate matching between video fine-granularity visual clues and text description is enhanced, and video features with different granularities are aligned, so that the cross-modal and cross-granularity alignment effect is improved, and the fine-granularity query requirement is met. A double-channel enhanced decoding module is adopted, and Mama and Transform are combined, so that a time retrieval task and a highlight detection task are optimized respectively, and the decoding precision and the task collaboration effect are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for decoupling fine-grained video moment retrieval and specular detection joint tasks. Background Technology

[0002] Video time-of-view retrieval: This task involves accurately locating a specific moment or segment from a video based on natural language descriptions. It requires understanding the temporal content of the video and the semantics of the text, overcoming the temporal complexity and computational challenges of long videos, and achieving semantic matching in multimodal interactions.

[0003] Video highlight detection, also known as video summarization, video essence extraction, or video highlight detection, aims to identify the most representative, exciting, or crucial segments from a video. The goal of this technology is to identify areas in a video that are particularly distinctive, possess special visual effects, or are the focus of attention.

[0004] Video feature modeling: Video feature modeling methods have evolved from manual feature extraction to deep learning modeling, and then to modern fine-grained feature modeling. Early manual methods included HOG and SIFT. In the era of deep learning, pre-trained models such as CLIP, Slowfast, C3D, I3D, and VGG have improved global semantic understanding capabilities.

[0005] Multimodal contrastive loss is a strategy used to improve the feature alignment quality of multimodal data (such as video and text). Through a contrastive learning framework, it enables the model to distinguish between positive sample pairs (such as query text descriptions and corresponding video clips) and negative sample pairs (such as query text and non-corresponding video clips), thereby optimizing the proximity of similar samples and the distance between different samples in the feature space.

[0006] Mamba: A linear time series modeling method based on a selective state-space model, used for efficient processing of long-sequence data.

[0007] This paper presents a Transformer for joint tasks of video time-of-motion retrieval and highlight detection. The core of this method lies in: designing a local-global multimodal alignment module to align video and text features; a query-guided visual feature refinement module to filter out irrelevant information; and task collaboration modules (HD2MR and MR2HD) to explicitly utilize the reciprocity between time-of-motion retrieval and highlight detection to optimize temporal localization and highlight score prediction, thereby achieving more accurate video content parsing. While this method considers inter-task reciprocity, it does not address the coupling issues during joint training. Pooling operations during alignment can introduce semantic ambiguity.

[0008] A modality-aware heterogeneous graph method for joint tasks of video moment retrieval and specular detection is proposed. Its core lies in capturing global contextual information and enhancing local relationships through two key modules: cross-modal statistical correlation interaction and heterogeneous graph inference, effectively handling the complexity of multimodal data. Furthermore, a lightweight joint task solution is designed, which not only significantly improves the model's accuracy and inference efficiency but also achieves lightweight design by reducing the number of parameters. This method requires constructing N×N node relationships (N being the number of video segments) during heterogeneous graph inference. As the video length increases, the computational complexity increases quadratically, and the understanding of the video becomes increasingly ambiguous.

[0009] A query-dependent Transformer for joint tasks of video moment retrieval and specular detection is proposed. Its core lies in constructing a video representation of query dependencies. Textual query information is explicitly injected into the video feature extraction process through a cross-attention mechanism. Simultaneously, negative sample pairs are used to enhance the model's utilization of query-related information, and an input adaptive saliency predictor is introduced to dynamically adjust the saliency scoring criteria. This allows for more accurate localization of key segments in the video and evaluation of their relevance to the query. One of the core innovations of this method is negative sample learning, where negative samples consist of segments that "do not match the query." However, when these segments are too similar to matching segments (video segments are highly similar), the model cannot learn subtle discriminative features, presenting a potential vulnerability.

[0010] Existing models suffer from a lack of fine-grained modeling. They mainly extract global visual and motion features from videos using pre-trained models such as CLIP, Slowfast, C3D, I3D, and VGG, but lack fine-grained video feature modeling strategies, resulting in insufficient ability to capture local entity relationships and spatiotemporal details.

[0011] Existing models suffer from coarse cross-modal alignment. In the process of cross-modal semantic alignment, they are usually matched only with the global features of the video and the global text, ignoring the precise alignment between semantic information of different granularities within the video and between information of different granularities and the text description, which makes it difficult to meet the needs of fine-grained queries.

[0012] Existing models suffer from a lack of task-specific modeling. They ignore the differences in focus between different subtasks during the decoding and prediction stages, and lack independent optimization for different subtasks. This can easily lead to information interference and performance trade-offs, limiting the collaborative optimization effect of joint tasks. Summary of the Invention

[0013] To address the issues of lack of fine-grained modeling, coarse cross-modal alignment, and lack of task-specific modeling in existing models, this invention provides a decoupling method for the joint task of fine-grained video moment retrieval and specular detection.

[0014] To achieve the above objectives, the present invention employs the following technical solutions:

[0015] A fine-grained, driven method for decoupling the joint tasks of video moment retrieval and specular detection, the method comprising the following steps:

[0016] Step 1: Establish a fine-grained, decoupled model for the joint task of video moment retrieval and highlight detection. Input the video and query text into the model to perform multi-granularity video feature extraction and query text feature extraction, and obtain object-granular features after dimensionality transformation. Video granularity features Describing granularity characteristics Query text features ;

[0017] The specific operation of step 1 is as follows:

[0018] Step 1.1: For a video, sampling is performed at 2-second intervals, and the video is input into a fine-grained video moment retrieval and specular detection joint task decoupling model for multi-granularity video feature extraction.

[0019] Object-level feature extraction process: The YOLOv8 object detection network, pre-trained on the COCO dataset, detects objects in video frames. The top 5 objects with the highest detection accuracy are extracted; if fewer than 5 are detected, zeros are padded. The CLIP visual encoder encodes the features of the 5 extracted objects and stacks them to form the object feature representation for that frame.

[0020] ,

[0021] in, Represents the object characteristics in frame t. Represents the feature of the nth object in frame t. , Represents the characteristic dimension of the object;

[0022] During data loading, the features of the five objects in each frame are concatenated into a one-dimensional vector, representing the object features of that frame. The formula is as follows: ,

[0023] in, This represents the object characteristics at frame t during data loading; therefore, the object granularity characteristics are:

[0024] ,

[0025] in, The number of video frames. Represents the granularity characteristics of the object.

[0026] Video granular feature extraction process: The CLIP visual encoder and Slowfast dynamic feature extraction network are combined to extract the static and dynamic visual features of the entire video. During loading, the static and dynamic visual features of the video are concatenated and combined to form the video granular features. ,in, Represents the dimension of video granularity features;

[0027] The descriptive granularity feature extraction process is as follows: A pre-trained Blip-2 graph-to-text network is used to generate natural language descriptions for video frames. Each frame's natural language description is then saved line by line to a text file. A pre-trained CLIP text encoder is then used to extract the semantic features of each line of natural language description. These features are then stacked temporally to construct the descriptive granularity features for the entire video. ,in, This represents the dimension of the descriptive granularity features;

[0028] After feature extraction, the corresponding feature mapping layer is used to transform the dimensionality of different features. The feature mapping layer consists of three receptive layers, uniformly mapping to 256 dimensions, as shown in the formula:

[0029] ,

[0030] ,

[0031] ,

[0032] in, These represent video features of different granularities after dimensionality transformation. These represent the corresponding three-layer perceptrons used;

[0033] Step 1.2: Input the query text into the fine-grained driven video moment retrieval and specular detection joint task decoupling model for query text feature extraction, and obtain the query text features after dimensionality transformation. ;

[0034] For the query text, the pre-trained CLIP text encoder is used to extract query text features. ,in, The feature of the i-th character in the current query text , This represents the number of characters in the current query. To determine the dimensionality of the query text features, a three-layer perceptron is used after feature extraction to perform a dimensionality transformation on the query text features, mapping them to 256 dimensions.

[0035] ,

[0036] in, The query text features are those after dimensional transformation. This represents the three-layer perceptron used for feature dimension transformation of the query text.

[0037] Step 2: Utilize local alignment loss and global alignment loss to refine object granular features Video granularity features Describing granularity characteristics Query text features Alignment, resulting in multi-granularity feature alignment loss. ;

[0038] The specific operation of step 2 is as follows:

[0039] Multi-granularity feature alignment loss is divided into two parts: local alignment loss and global alignment loss; it incorporates three different granularities of video features. , , and query text features The input is fed into the multi-granularity feature alignment loss to perform local alignment loss and global alignment loss operations;

[0040] The local alignment loss process is as follows: For the input video features and query text features, L2 feature normalization is first performed. Then, a similarity matrix between frames and words, objects, or descriptions is obtained by calculating the similarity matrix between the two features. This matrix measures the matching degree between the two features. Subsequently, padding words or frames are ignored using a mask. The frame-level matching score between the most relevant text words, objects, or descriptions in each frame is extracted from the similarity matrix. Invalid similarities are filtered using a video mask, and the distribution sharpness is controlled by a temperature coefficient to obtain the final logits value. Finally, the binary cross-entropy loss function is used to force the distinction between positive and negative samples.

[0041]

[0042] in, This represents the local alignment loss in multi-granularity feature alignment loss; Where T is the batch size, and T is the number of video frames in the video features. This represents the encoding of the positively correlated sample locations in the video. The sigmoid activation function is given by the formula: ; Indicates batch size; Indicates the relative position of the current frame;

[0043] For the local alignment loss, local alignment is performed between the three granularities of video features and between the three granularities of video features and the query text. The final overall local alignment loss is calculated as follows. The expression is:

[0044] ;

[0045] in, The weights for the local alignment loss, This represents the local alignment loss between video-level features and query text features. This represents the local alignment loss between object-level features and query text features. This represents the local alignment loss between descriptive granular features and query text features. This represents the local alignment loss between the granular features and the object's granular features. This represents the local alignment loss between object-level granular features and video-level granular features;

[0046] By calculating video granularity features Describing granularity characteristics Features of query text Global contrast loss between the two establishes macroscopic semantic relationships between video and text. : ,

[0047] in, , Each is the first in a batch Global features of the first video and the second The global features of a query text, where B is the batch size. Indicates the first Global features of a query text Indicates the transpose operation;

[0048] The global alignment loss aligns video-level features and query text features, as well as description-level features and query text features. The expression for the overall global alignment loss is:

[0049] ,

[0050] in, The weights for the global alignment loss. The loss represents the alignment loss between video-level features and query text features. This represents the alignment loss between descriptive granular features and query text features;

[0051] Finally, the local alignment loss and the global alignment loss are added together to form the complete multi-granularity feature alignment loss. The overall expression for the multi-granularity feature alignment loss is as follows: .

[0052] Step 3: Transform the video features of three different granularities output in Step 1. , , and query text features after dimensional transformation The input is fed into the video feature refinement module for cross-modal interaction. Then, the three granularity features are fed into the gated adaptive fusion module for fusion, resulting in multi-granularity video features fused from the three different granularity video features. Finally, the video feature refinement module is used to refine the video features. Cross-modal interaction with query text features to obtain ;

[0053] The specific operation of step 3 is as follows:

[0054] A gated adaptive fusion module is used to fuse video features of different granularities. Before and after fusion, the video feature refinement method in TR-DETR is adopted. Before and after fusion, video features of different granularities and the fused multi-granularity video features are processed through different refinement modules to achieve cross-modal interaction with the query text. , , ;

[0055] In the gated adaptive fusion module, pairwise feature fusion is achieved through a gating unit. The specific operation of the gating unit is as follows: first, the two input feature tensors are concatenated along the last dimension.

[0056] ,

[0057] in, Representative feature splicing operation, These are the two features input. To facilitate the subsequent computation of the one-dimensional convolution in the real dimension T, we will concatenate the tensor... Perform a dimension permutation to change the dimensional order of the tensor from [B, T, 2D] to [B, 2D, T]: ,

[0058] in, This represents a dimension permutation operation; the subsequent one-dimensional convolution operation generates an initial gating signal based on the joint information of the two features: ,

[0059] in, This represents a one-dimensional convolution operation, after which the gated signal is restored to its original dimension to facilitate element-wise multiplication with the original features. Then, the sigmoid activation function is used to compress its value to between 0 and 1:

[0060] .

[0061] Finally, the obtained gating weights are used to perform a weighted summation of the two original input features E and H to obtain the final fused features:

[0062] ,

[0063] in, This represents element-wise multiplication;

[0064] Step 3.1: Combine video features at three different granularities , , The multi-granularity video feature refinement module input into the model and the query text features are obtained through cross-modal interaction with the query text via different refinement modules. , , ;

[0065] Step 3.2: The gated adaptive fusion module in the multi-granularity feature fusion module of the model processes the features after feature refinement at three different granularities. , , Each part is encoded independently, and three independent granular encoders composed of Transformer encoders are used to extract the high-level semantic representation of each granularity.

[0066] Then, the encoded video granularity features and description granularity features are weighted and fused through the first gate unit in the model-gated adaptive fusion module. The fused features are then fed into the Transformer encoder to obtain the fused video granularity and description granularity features. ;

[0067] Then, the encoded object granular features and The fusion is performed using the second gating unit in the model-gated adaptive fusion module to obtain multi-granularity video features with semantic supplementation and encoding. ;

[0068] Step 3.3: Utilize the video feature refinement module to refine the multi-granularity video features obtained in Step 3.2 after semantic supplementation and encoding. Interact with text features to obtain .

[0069] Step 4: Combine the fused multi-granularity video features obtained in Step 3. The input is fed into the dual-path enhancement decoding module of the model. Using two independent sub-paths in the module, semantic specialization enhancement is performed on the video time-to-time retrieval and highlight detection tasks in the highlight detection branch and the video time-to-time retrieval branch, respectively, and finally the relevant time and highlight score are obtained.

[0070] In the highlight detection branch, the result obtained in step 3.3 is... The input is fed into a specular information enhancer based on a Transformer encoder, for Highlight information enhancement is performed to highlight the instantaneous dynamics and visual changes related to highlight events in the video features, and it is directly connected to the highlight prediction head to obtain the highlight score of each frame through linear projection and normalization processing.

[0071] In the video moment retrieval branch, the results obtained in step 3.3 are... Input to a Mamba-based time information enhancer, for Perform time information augmentation, and then apply the augmented time information... The input is fed to the time decoder for decoding, and the decoded text is then fed to the time prediction header, which outputs the time of the segment related to the query text.

[0072] Furthermore, in step 1, when extracting object-granular features, the object detection network can be any one of YOLOv8, Faster R-CNN, or SSD;

[0073] When extracting video granular features, any one of the following dynamic feature extraction networks can be selected: Slowfast, VGG, I3D, and C3D.

[0074] When extracting descriptive granularity features, any one of the following graph-based text networks can be selected: Blip-2, LLaVA, Qwen-VL, and Qwen2-VL.

[0075] Compared with the prior art, the present invention has the following advantages:

[0076] This invention employs a multi-granularity video feature modeling approach. Compared to traditional methods, this approach emphasizes key semantics within the video, effectively enhancing the richness and accuracy of video semantic expression. It addresses the shortcomings of existing methods in fine-grained alignment of video text and alignment between different granularities within the video. It also alleviates the coupling problem between video moment retrieval and highlight detection tasks when they are performed as a joint task. The modules designed and utilized in this invention are interconnected and interact with each other, improving model prediction accuracy and generating prediction results that are more consistent with reality. Attached Figure Description

[0077] Figure 1A technical framework diagram for a fine-grained, video moment retrieval and specular detection joint task decoupling method. Detailed Implementation

[0078] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0079] A fine-grained, driven method for decoupling the joint tasks of video moment retrieval and specular detection, characterized in that the method includes the following steps:

[0080] Step 1: Establish a fine-grained, decoupled model for the joint task of video moment retrieval and highlight detection. Input the video and query text into the model to perform multi-granularity video feature extraction and query text feature extraction, and obtain object-granular features after dimensionality transformation. Video granularity features Describing granularity characteristics Query text features ;

[0081] The specific operation of step 1 is as follows:

[0082] Step 1.1: For a video, sampling is performed at 2-second intervals, and the video is input into a fine-grained video moment retrieval and specular detection joint task decoupling model for multi-granularity video feature extraction.

[0083] Object-level feature extraction process: The YOLOv8 object detection network, pre-trained on the COCO dataset, detects objects in video frames. The top 5 objects with the highest detection accuracy are extracted; if fewer than 5 are detected, zeros are padded. The CLIP visual encoder encodes the features of the 5 extracted objects and stacks them to form the object feature representation for that frame.

[0084] ,

[0085] in, Represents the object characteristics in frame t. Represents the feature of the nth object in frame t. , Represents the characteristic dimension of the object;

[0086] During data loading, the features of the five objects in each frame are concatenated into a one-dimensional vector, representing the object features of that frame. The formula is as follows: ,

[0087] in, This represents the object characteristics at frame t during data loading; therefore, the object granularity characteristics are:

[0088] ,

[0089] in, The number of video frames. Represents the granularity characteristics of the object.

[0090] Video granular feature extraction process: The CLIP visual encoder and Slowfast dynamic feature extraction network are combined to extract the static and dynamic visual features of the entire video. During loading, the static and dynamic visual features of the video are concatenated and combined to form the video granular features. ,in, Represents the dimension of video granularity features;

[0091] The descriptive granularity feature extraction process is as follows: A pre-trained Blip-2 graph-to-text network is used to generate natural language descriptions for video frames. Each frame's natural language description is then saved line by line to a text file. A pre-trained CLIP text encoder is then used to extract the semantic features of each line of natural language description. These features are then stacked temporally to construct the descriptive granularity features for the entire video. ,in, This represents the dimension of the descriptive granularity features;

[0092] After feature extraction, the corresponding feature mapping layer is used to transform the dimensionality of different features. The feature mapping layer consists of three receptive layers, uniformly mapping to 256 dimensions, as shown in the formula:

[0093] ,

[0094] ,

[0095] ,

[0096] in, These represent video features of different granularities after dimensionality transformation. These represent the corresponding three-layer perceptrons used;

[0097] Step 1.2: Input the query text into the fine-grained driven video moment retrieval and specular detection joint task decoupling model for query text feature extraction, and obtain the query text features after dimensionality transformation. ;

[0098] For the query text, the pre-trained CLIP text encoder is used to extract query text features. ,in, The feature of the i-th character in the current query text , This represents the number of characters in the current query. To determine the dimensionality of the query text features, a three-layer perceptron is used after feature extraction to perform a dimensionality transformation on the query text features, mapping them to 256 dimensions.

[0099] ,

[0100] in, The query text features are those after dimensional transformation. This represents the three-layer perceptron used for feature dimension transformation of the query text.

[0101] Step 2: Utilize local alignment loss and global alignment loss to refine object granular features Video granularity features Describing granularity characteristics Query text features Alignment, resulting in multi-granularity feature alignment loss. ;

[0102] The specific operation of step 2 is as follows:

[0103] Multi-granularity feature alignment loss is divided into two parts: local alignment loss and global alignment loss; it incorporates three different granularities of video features. , , and query text features The input is fed into the multi-granularity feature alignment loss to perform local alignment loss and global alignment loss operations;

[0104] The local alignment loss process is as follows: For the input video features and query text features, L2 feature normalization is first performed. Then, a similarity matrix between frames and words, objects, or descriptions is obtained by calculating the similarity matrix between the two features. This matrix measures the matching degree between the two features. Subsequently, padding words or frames are ignored using a mask. The frame-level matching score between the most relevant text words, objects, or descriptions in each frame is extracted from the similarity matrix. Invalid similarities are filtered using a video mask, and the distribution sharpness is controlled by a temperature coefficient to obtain the final logits value. Finally, the binary cross-entropy loss function is used to force the distinction between positive and negative samples.

[0105]

[0106] in, This represents the local alignment loss in multi-granularity feature alignment loss; Where T is the batch size, and T is the number of video frames in the video features. This represents the encoding of the positively correlated sample locations in the video. The sigmoid activation function is given by the formula: ; Indicates batch size; Indicates the relative position of the current frame;

[0107] For the local alignment loss, local alignment is performed between the three granularities of video features and between the three granularities of video features and the query text. The final overall local alignment loss is calculated as follows. The expression is:

[0108] ;

[0109] in, The weights for the local alignment loss, This represents the local alignment loss between video-level features and query text features. This represents the local alignment loss between object-level features and query text features. This represents the local alignment loss between descriptive granular features and query text features. This represents the local alignment loss between the granular features and the object's granular features. This represents the local alignment loss between object-level granular features and video-level granular features;

[0110] By calculating video granularity features Describing granularity characteristics Features of query text Global contrast loss between the two establishes macroscopic semantic relationships between video and text. : ,

[0111] in, , Each is the first in a batch Global features of the first video and the first The global features of a query text, where B is the batch size. Indicates the first Global features of a query text Indicates the transpose operation;

[0112] The global alignment loss aligns video-level features and query text features, as well as description-level features and query text features. The expression for the overall global alignment loss is:

[0113] ,

[0114] in, The weights for the global alignment loss. The loss represents the alignment loss between video-level features and query text features. This represents the alignment loss between descriptive granular features and query text features;

[0115] Finally, the local alignment loss and the global alignment loss are added together to form the complete multi-granularity feature alignment loss. The overall expression for the multi-granularity feature alignment loss is as follows: .

[0116] Step 3: Transform the video features of three different granularities output in Step 1. , , and query text features after dimensional transformation The input is fed into the video feature refinement module for cross-modal interaction. Then, the three granularity features are fed into the gated adaptive fusion module for fusion, resulting in multi-granularity video features fused from the three different granularity video features. Finally, the video feature refinement module is used to refine the video features. Cross-modal interaction with query text features to obtain ;

[0117] The specific operation of step 3 is as follows:

[0118] A gated adaptive fusion module is used to fuse video features of different granularities. Before and after fusion, the video feature refinement method in TR-DETR is adopted. Before and after fusion, video features of different granularities and the fused multi-granularity video features are processed through different refinement modules to achieve cross-modal interaction with the query text. , , ;

[0119] In the gated adaptive fusion module, pairwise feature fusion is achieved through a gating unit. The specific operation of the gating unit is as follows: first, the two input feature tensors are concatenated along the last dimension.

[0120] ,

[0121] in, Representative feature splicing operation, These are the two features input. To facilitate the subsequent computation of the one-dimensional convolution in the real dimension T, we will concatenate the tensor... Perform a dimension permutation to change the dimensional order of the tensor from [B, T, 2D] to [B, 2D, T]: ,

[0122] in, This represents a dimension permutation operation; the subsequent one-dimensional convolution operation generates an initial gating signal based on the joint information of the two features: ,

[0123] in, This represents a one-dimensional convolution operation, after which the gated signal is restored to its original dimension to facilitate element-wise multiplication with the original features. Then, the sigmoid activation function is used to compress its value to between 0 and 1:

[0124] .

[0125] Finally, the obtained gating weights are used to perform a weighted summation of the two original input features E and H to obtain the final fused features:

[0126] ,

[0127] in, This represents element-wise multiplication;

[0128] Step 3.1: Combine video features at three different granularities , , The multi-granularity video feature refinement module input into the model and the query text features are obtained through cross-modal interaction with the query text via different refinement modules. , , ;

[0129] Step 3.2: The gated adaptive fusion module in the multi-granularity feature fusion module of the model processes the features after feature refinement at three different granularities. , , Each part is encoded independently, and three independent granular encoders composed of Transformer encoders are used to extract the high-level semantic representation of each granularity.

[0130] Then, the encoded video granular features and description granular features are weighted and fused through the first gate unit in the model-gated adaptive fusion module. The fused features are then fed into the Transformer encoder to obtain the fused video granular and description granular features. ;

[0131] Afterwards, the encoded object granular features and The fusion is performed using the second gating unit in the model-gated adaptive fusion module to obtain multi-granularity video features with semantic supplementation and encoding. ;

[0132] Step 3.3: Utilize the video feature refinement module to refine the multi-granularity video features obtained in Step 3.2 after semantic supplementation and encoding. Interact with text features to obtain .

[0133] Step 4: Combine the fused multi-granularity video features obtained in Step 3. The input is fed into the dual-path enhancement decoding module. Using two independent sub-paths in the module, semantic specialization enhancement is performed on the video time retrieval and highlight detection tasks in the highlight detection branch and the video time retrieval branch, respectively, to finally obtain the relevant time and highlight score.

[0134] In the highlight detection branch, the result obtained in step 3.3 is... The input is fed into a specular information enhancer based on a Transformer encoder, for Highlight information enhancement is performed to highlight the instantaneous dynamics and visual changes related to highlight events in the video features, and it is directly connected to the highlight prediction head to obtain the highlight score of each frame through linear projection and normalization processing.

[0135] In the video moment retrieval branch, the results obtained in step 3.3 are... Input to a Mamba-based time information enhancer, for Perform time information augmentation, and then apply the augmented time information... The input is fed to the time decoder for decoding, and the decoded text is then fed to the time prediction header, which outputs the time of the segment related to the query text.

[0136] Table 1 shows the performance comparison of the proposed method with mainstream methods in recent years on the QVHighlights dataset. Specifically, in the video moment retrieval task, the proposed method achieves an R1@0.5 score of 68.09, a 2.66 improvement over the second-best method CG-DETR (65.43); and an R1@0.7 score of 49.94, a 0.98 improvement over the second-best method TR-DETR (48.96). Under the more comprehensive mAP evaluation framework, the advantages of the proposed method are even more pronounced: mAP@0.5 is 66.19 (an improvement of 1.68), mAP@0.75 is 45.09 (an improvement of 1.36), and the average mAP is 45.01 (an improvement of 1.83). Furthermore, in the video highlight detection task, the proposed method also achieves an mAP (≥Very Good) score of 41.83, a 1.5 improvement over the second-best method. These data consistently demonstrate the effectiveness and advancement of the proposed method.

[0137] Table 1 Performance comparison between mainstream methods and this method

[0138]

[0139] Note: The underlined part is the second best, and the bolded part is the best.

[0140] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A fine-grained, driven method for decoupling the joint tasks of video time-of-motion retrieval and specular detection, characterized in that, The method includes the following steps: Step 1: Establish a fine-grained, decoupled model for the joint task of video moment retrieval and highlight detection. Input the video and query text into the model to perform multi-granularity video feature extraction and query text feature extraction, and obtain object-granular features after dimensionality transformation. Video granularity features Describing granularity characteristics Query text features ; Step 2: Utilize local alignment loss and global alignment loss to refine object granular features Video granularity features Describing granularity characteristics Query text features Alignment, resulting in multi-granularity feature alignment loss. ; Step 3: Transform the video features of three different granularities output in Step 1. , , and query text features after dimensional transformation The input is fed into the video feature refinement module for cross-modal interaction. Then, the three granularity features are fed into the gated adaptive fusion module for fusion, resulting in multi-granularity video features fused from the three different granularity video features. Finally, the video feature refinement module is used to refine the video features. Cross-modal interaction with query text features to obtain ; Step 4: Combine the fused multi-granularity video features obtained in Step 3. The input is fed into the dual-path enhancement decoding module of the model. Using two independent sub-paths in the module, semantic specialization enhancement is performed on the video time-to-time retrieval and highlight detection tasks in the highlight detection branch and the video time-to-time retrieval branch, respectively, and finally the relevant time and highlight score are obtained.

2. The fine-grained driven video moment retrieval and specular detection joint task decoupling method according to claim 1, characterized in that, The specific operation of step 1 is as follows: Step 1.1: For a video, sampling is performed at 2-second intervals, and the video is input into a fine-grained video moment retrieval and specular detection joint task decoupling model for multi-granularity video feature extraction. Object-level feature extraction process: The YOLOv8 object detection network, pre-trained on the COCO dataset, detects objects in video frames. The top 5 objects with the highest detection accuracy are extracted; if fewer than 5 are detected, zeros are padded. The CLIP visual encoder encodes the features of the 5 extracted objects and stacks them to form the object feature representation for that frame. ， in, Represents the object characteristics in frame t. Represents the feature of the nth object in frame t. , Represents the characteristic dimension of the object; During data loading, the features of the five objects in each frame are concatenated into a one-dimensional vector, representing the object features of that frame. The formula is as follows: , in, This represents the object characteristics at frame t during data loading; therefore, the object granularity characteristics are: ， in, The number of video frames. Represents the granularity characteristics of the object; Video granular feature extraction process: The CLIP visual encoder and Slowfast dynamic feature extraction network are combined to extract the static and dynamic visual features of the entire video. During loading, the static and dynamic visual features of the video are concatenated and combined to form the video granular features. ,in, Represents the dimension of video granularity features; The descriptive granularity feature extraction process is as follows: A pre-trained Blip-2 graph-to-text network is used to generate natural language descriptions for video frames. Each frame's natural language description is then saved line by line to a text file. A pre-trained CLIP text encoder is then used to extract the semantic features of each line of natural language description. These features are then stacked temporally to construct the descriptive granularity features for the entire video. ,in, This represents the dimension of the descriptive granularity features; After feature extraction, the corresponding feature mapping layer is used to transform the dimensionality of different features. The feature mapping layer consists of three receptive layers, uniformly mapping to 256 dimensions, as shown in the formula: ， ， ， in, These represent video features of different granularities after dimensionality transformation. These represent the corresponding three-layer perceptrons used; Step 1.2: Input the query text into the fine-grained driven video moment retrieval and specular detection joint task decoupling model for query text feature extraction, and obtain the query text features after dimensionality transformation. ; For the query text, the pre-trained CLIP text encoder is used to extract query text features. ,in, The feature of the i-th character in the current query text , This represents the number of characters in the current query. To determine the dimensionality of the query text features, a three-layer perceptron is used after feature extraction to perform a dimensionality transformation on the query text features, mapping them to 256 dimensions. ， in, The query text features are those after dimensional transformation. This represents the three-layer perceptron used for feature dimension transformation of the query text.

3. The fine-grained driven video moment retrieval and specular detection joint task decoupling method according to claim 2, characterized in that, The specific operation of step 2 is as follows: Multi-granularity feature alignment loss is divided into two parts: local alignment loss and global alignment loss; it incorporates three different granularities of video features. , , and query text features The input is fed into the multi-granularity feature alignment loss to perform local alignment loss and global alignment loss operations; The local alignment loss process is as follows: For the input video features and query text features, L2 feature normalization is first performed. Then, the similarity matrix between frames and words, objects, or descriptions is obtained by calculating the similarity matrix between the two features to measure the matching degree between the two features. Subsequently, padding words or frames are ignored using a mask. The frame-level matching score between the most relevant text words, objects, or descriptions in each frame is extracted from the similarity matrix. Invalid similarities are filtered using a video mask, and the distribution sharpness is controlled by a temperature coefficient to obtain the final logits value. Finally, the binary cross-entropy loss function is used to force the distinction between positive and negative samples. ， in, This represents the local alignment loss in multi-granularity feature alignment loss; Where T is the batch size, and T is the number of video frames in the video features. This represents the encoding of positively correlated sample locations in the video. The sigmoid activation function is given by the formula: ; Indicates batch size; Indicates the relative position of the current frame; For the local alignment loss, local alignment is performed between the three granularities of video features and between the three granularities of video features and the query text. The final overall local alignment loss is calculated as follows. The expression is: ； in, The weights for the local alignment loss, This represents the local alignment loss between video-level features and query text features. This represents the local alignment loss between object-level features and query text features. This represents the local alignment loss between descriptive granular features and query text features. This represents the local alignment loss between the granular features and the object's granular features. This represents the local alignment loss between object-level granular features and video-level granular features; By calculating video granularity features Describing granularity characteristics Features of query text Global contrast loss between the two establishes macroscopic semantic relationships between video and text. : ， in, , Each is the first in a batch Global features of the first video and the second The global features of a query text, where B is the batch size. Indicates the first Global features of a query text Indicates the transpose operation; The global alignment loss aligns video-level features and query text features, as well as description-level features and query text features. The expression for the overall global alignment loss is: ， in, The weights for the global alignment loss. The loss represents the alignment loss between video-level features and query text features. This represents the alignment loss between descriptive granular features and query text features; Finally, the local alignment loss and the global alignment loss are added together to form the complete multi-granularity feature alignment loss. The overall expression for the multi-granularity feature alignment loss is as follows: .

4. The fine-grained driven video moment retrieval and specular detection joint task decoupling method according to claim 3, characterized in that, The specific operation of step 3 is as follows: Step 3.1: Combine video features at three different granularities , , The multi-granularity video feature refinement module input into the model and the query text features are obtained through cross-modal interaction with the query text via different refinement modules. , , ; Step 3.2: The gated adaptive fusion module in the multi-granularity feature fusion module of the model processes the features after feature refinement at three different granularities. , , Each part is encoded independently, and three independent granular encoders composed of Transformer encoders are used to extract the high-level semantic representation of each granularity. Then, the encoded video granular features and description granular features are weighted and fused through the first gate unit in the model-gated adaptive fusion module. The fused features are then fed into the Transformer encoder to obtain the fused video granular and description granular features. ; Afterwards, the encoded object granular features and The fusion is performed using the second gating unit in the model-gated adaptive fusion module to obtain multi-granularity video features with semantic supplementation and encoding. ; Step 3.3: Utilize the video feature refinement module to refine the multi-granularity video features obtained in Step 3.2 after semantic supplementation and encoding. Interact with text features to obtain .

5. The fine-grained driven video moment retrieval and specular detection joint task decoupling method according to claim 4, characterized in that, The specific operation of step 4 is as follows: In the highlight detection branch, the result obtained in step 3.3 is... The input is fed into a specular information enhancer based on a Transformer encoder, for Highlight information enhancement is performed to highlight the instantaneous dynamics and visual changes related to highlight events in the video features, and it is directly connected to the highlight prediction head to obtain the highlight score of each frame through linear projection and normalization processing. In the video moment retrieval branch, the results obtained in step 3.3 are... Input to a Mamba-based time information enhancer, for Perform time information augmentation, and then apply the augmented time information... The input is fed to the time decoder for decoding, and the decoded text is then fed to the time prediction header, which outputs the time of the segment related to the query text.

6. The fine-grained driven video moment retrieval and specular detection joint task decoupling method according to claim 5, characterized in that, In step 1, when extracting object granular features, the object detection network can be any one of YOLOv8, Faster R-CNN, or SSD. When extracting video granular features, any one of the following dynamic feature extraction networks can be selected: Slowfast, VGG, I3D, and C3D. When extracting descriptive granularity features, any one of the following graph-based text networks can be selected: Blip-2, LLaVA, Qwen-VL, and Qwen2-VL.

Images