Dual-granularity alignment efficient partial correlation video retrieval based on implicit fragment modeling and semantic decomposition

Abstract

The invention provides double-granularity alignment efficient partial correlation video retrieval based on implicit fragment modeling and semantic decomposition, and aims to solve the problems of information redundancy and low efficiency in an existing video modeling method, the problem of granularity mismatching between sentence representation and video frame features and the problem that alignment between a text and a video is not refined enough. According to the method, the expression and modeling of video data are optimized by introducing a structure combining a Gaussian mixture model and Transform, and multi-scale local details with short time span are adaptively integrated by introducing a window attention mechanism and a cross attention mechanism, so that finer features are obtained, and cross-modal similarity score calculation of texts and videos is facilitated. The method comprises the following steps: data preprocessing and frame segmentation: preprocessing and segmenting an input video into frames, and preparing for feature extraction after each frame of image is subjected to standardization processing; implicit modeling is carried out on the fragment-level features, uniformly sampled frame-level visual features are input into a Gaussian mixture modeling module, adjacent frame focusing modeling is carried out through a multi-scale Gaussian attention mechanism, and fragment-level representation of different receptive fields is implicitly formed. Enhancing local details of the frame-level features, performing convolution operation on each frame by adopting windows with different scales, and calculating a feature relationship in each local window; local features of all scales are fused through a cross attention mechanism, semantic expression of frame features is enhanced, and it is ensured that each video frame can capture important local details.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the field of computer vision, and particularly relates to a dual-granularity alignment efficient partial relevant video retrieval based on implicit segment modeling and semantic decomposition. BACKGROUND

[0002] With the explosive growth of global video data, how to efficiently and accurately extract the content required by users from massive videos has become one of the core problems in the field of information retrieval. In the Internet, social media, monitoring systems, online education and other scenarios, the surge of video data has brought challenges of information overload. According to statistics, the online video content generated globally every day reached millions of hours in 2022, and the traditional retrieval method based on text tags has been unable to meet the complex search needs of users. Users often do not have the ability to accurately describe the content of the video, which makes the retrieval method relying only on the video title, tag or metadata have serious limitations in accuracy.

[0003] Since ordinary users prefer to express their information needs through natural language queries, the research on text-to-video retrieval (T2VR) is very important. Given a query in the form of a natural language sentence, T2VR requires retrieving videos from a video library that are semantically relevant to the given query. Current T2VR methods are trained and tested on video caption-oriented datasets such as MSVD, MSRVTT and VATEX. A key characteristic of these datasets is that videos are assumed to be pre-cut in a short time, and the provided captions well describe the highlights of the video content. Therefore, for a given pair of video and caption, the video should be completely relevant to the caption. However, in reality, due to the query beforehand, the pre-cut video clip may not contain enough content to fully meet the query. This shows the gap between literature and the real world. Partially relevant video retrieval (PRVR) can help users directly locate the relevant segments in long videos through precise matching of text and video segments. This technology not only reduces the time cost of users screening irrelevant information, but also improves the accuracy of video retrieval and user experience. Unlike traditional retrieval, this method requires in-depth analysis of video content to ensure high relevance between specific segments of the video and the text description. SUMMARY

[0004] The purpose of the present application is to overcome the problems of information redundancy and large storage overhead of existing PRVR explicit modeling, as well as the problems of mismatch between sentence representation and video frame features and insufficient fine-grained alignment between text and video. We propose a new dual-granularity alignment efficient partial relevant video retrieval network architecture based on implicit segment modeling and semantic decomposition.

[0005] The present application aims at the information redundancy problem and the low efficiency problem of explicit modeling, and proposes an implicit clipping representation modeling transformer based on Gaussian mixture model, which has the advantages of multi-scale Gaussian constraint and high information density, compact segment embedding. It focuses on adjacent frames instead of the whole video for each frame. Thus, efficient local modeling and reduction of information redundancy are realized. The specific manufacturing method comprises the following steps:

[0006] (1) Data preprocessing and frame segmentation, respectively using a pre-trained visual encoder to extract multi-modal visual features from the video frame sequence, including using a two-dimensional convolutional neural network (2D CNN) to extract image appearance features and using a three-dimensional convolutional neural network (3D CNN) to extract temporal motion features.

[0007] (2) Implicit modeling of segment-level features, inputting uniformly sampled frame-level visual features into a Gaussian mixture modeling module, and focusing on adjacent frames through a multi-scale Gaussian attention mechanism. Here, the input features are weighted to capture the dependency between features, and an implicit segment-level representation of different receptive fields is formed.

[0008] (3) Feature enhancement, the output of the Gaussian attention module is further processed through a feedforward neural network to further extract and combine features. At the same time, residual connections are used between the Gaussian attention module and the feedforward network, which can prevent gradient disappearance and help the model better perform deep training.

[0009] (4) Feature fusion, multiple Gaussian windows (low, medium, high, and infinite variance) are used in parallel to generate segment embeddings of different scales, and then averaged to form a unified multi-scale fusion representation, enhancing the modeling ability of different duration target segments.

[0010] (5) Similarity matching calculation calculates the matching degree between text and video, and calculates the maximum similarity (local matching) between sentence vectors and multiple segment representations.

[0011] The present application aims at the problem of granularity mismatch between sentence representation and video frame features, and the problem that frame-level features cannot capture local details. The video features use a window attention mechanism combined with a cross-attention mechanism to adaptively integrate multi-scale local details of short time span, obtain more detailed features, and the frame branch uses semantic decomposition matching to decompose the sentence into multiple semantic units, thereby aligning with the frame features with enhanced local details. The specific manufacturing method comprises the following steps:

[0012] (1) Multi-scale window attention mechanism, using a window attention mechanism to divide the video frames into multiple small windows, and weighting and aggregating the features in each window. This process is achieved through a window mask to focus on local areas in the video.

[0013] (2) Adaptive feature aggregation, using adaptive aggregation to aggregate local features of different windows into global features. Through multi-scale window processing, the model can capture local details of the video at different scales.

[0014] (3) Cross-attention mechanism, through cross-attention, each frame can adaptively select and aggregate the most relevant information from its surrounding context of different scales.

[0015] (4) Fully connected layer (FC), further processing the aggregated local features through the fully connected layer to obtain more accurate matching representation. This feature is used to calculate the similarity between text and video features.

[0016] (5) Semantic decomposition matching, decomposing the entire sentence query into multiple fine-grained, frame-level semantic units (query vectors), aligning the granularity of the text with the granularity of the video frames.

[0017] (6) Similarity matching calculation, calculating the similarity between each semantic unit and all frames, finding the maximum similarity score of each semantic unit and all frames, and finally averaging to obtain the score of the frame branch. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is the total model diagram of the present application;

[0020] Figure 2 is the local modeling model diagram of the present application using Gaussian mixture model;

[0021] Figure 3 is the model diagram for calculating the similarity between text features and video features of the present application; DETAILED DESCRIPTION

[0022] The specific embodiments of the present application will be described below in conjunction with the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0023] For details, see Figure 1 Dual-granularity alignment efficient partial relevant video retrieval based on implicit segment modeling and semantic decomposition, comprising the steps of:

[0024] S1: Data preprocessing. Frame-level sampling is performed on each training video sample. If the number of frames exceeds the set upper limit, uniform downsampling is performed to standardize the input length.

[0025] S2: Visual feature extraction, using a pre-trained visual encoder to extract multimodal visual features from the video frame sequence, including using a two-dimensional convolutional neural network (2D CNN) to extract image appearance features and using a three-dimensional convolutional neural network (3D CNN) to extract temporal motion features.

[0026] S21: Use ResNet152 as a 2D CNN to obtain static appearance features for each frame of the image.

[0027] S22: Use I3D as a 3D CNN to obtain dynamic motion features of multi-frame sequences.

[0028] S3: Text feature encoding. Input the text query into a pre-trained language model (such as RoBERTa), extract word-level representations, process them through position encoding and Transformer, and then aggregate them into sentence-level semantic vectors through an attention mechanism.

[0029] S4: Implicit modeling of fragment-level features involves inputting uniformly sampled frame-level visual features into the Gaussian mixture modeling module. A multi-scale Gaussian attention mechanism is used for adjacent frame focusing modeling, implicitly forming fragment-level representations with different receptive fields. This avoids the information redundancy and storage burden associated with explicit sliding window construction. The output of the Gaussian attention module is:

[0030]

[0031] The Gaussian matrix is ​​defined as follows:

[0032]

[0033] S5: Feature fusion uses multiple Gaussian windows (low, medium, high, infinite variance) in parallel to generate segment embeddings at different scales, and performs average fusion to form a unified multi-scale fusion representation, enhancing the ability to model target segments of different durations.

[0034] S6: Frame-level feature local detail enhancement. Convolution operations are performed on windows of different scales for each frame to calculate the feature relationships within each local window. Local features at various scales are fused through a cross-attention mechanism to enhance the semantic expression of frame features and ensure that important local details can be captured in each video frame.

[0035] S61: A multi-scale window attention mechanism that performs convolutional processing at different scales on each frame to extract video frame features from both local and global perspectives. The window attention mechanism can be described as follows:

[0036]

[0037] S62: Cross-attention mechanism, which cross-fuses the features of each local window through an attention mechanism to generate multi-scale enhanced features.

[0038] S63: Residual connections and nonlinear mappings further enhance the semantic expressive power of frame features through residual connections and nonlinear fully connected layers. The final frame features are:

[0039]

[0040] S7: Semantic decomposition matching breaks down sentences into multiple semantic units, each corresponding to frame-level semantics. Word features are grouped into learnable semantic units through a projection layer, Transformer, and Slot Attention, ensuring that each unit covers specific content from the sentence that might appear in the frame-level video. Furthermore, a Semantic Diversity Loss is introduced to encourage diversity among the generated semantic units.

[0041] S8: Similarity matching calculation, which calculates the matching degree between text and video, including the similarity between sentence vectors and the overall video representation, as well as the similarity between each semantic unit and all frames. Finally, the two are weighted and fused to form the final matching score.

[0042] S9: Training optimization, using multiple loss functions for training, including triplet ranking loss, information contrast loss and query diversity loss, to enhance the model's ability to express partial matching relationships and preserve semantic structure. In addition, there is frame semantic alignment loss, which is a novel loss function that not only aligns between video and text, but also decouples between video frames and semantic units.

Claims

1. A dual-granularity aligned, high-efficiency partially relevant video retrieval method based on implicit fragment modeling and semantic decomposition, characterized in that: Including the following steps: S1: Data preprocessing. Frame-level sampling is performed on each training video sample. If the number of frames exceeds the set upper limit, uniform downsampling is performed to standardize the input length. S2: Visual feature extraction, using a pre-trained visual encoder to extract multimodal visual features from the video frame sequence, including using a two-dimensional convolutional neural network (2D CNN) to extract image appearance features and using a three-dimensional convolutional neural network (3D CNN) to extract temporal motion features. S3: Text feature encoding. Input the text query into a pre-trained language model (such as RoBERTa), extract word-level representations, process them through position encoding and Transformer, and then aggregate them into sentence-level semantic vectors through an attention mechanism. S4: Implicit modeling of fragment-level features involves inputting uniformly sampled frame-level visual features into the Gaussian mixture modeling module. Adjacent frame focusing modeling is performed through a multi-scale Gaussian attention mechanism, implicitly forming fragment-level representations with different receptive fields. This avoids information redundancy and storage burden caused by explicit sliding window construction. S5: Feature fusion uses multiple Gaussian windows (low, medium, high, infinite variance) in parallel to generate segment embeddings at different scales, and performs average fusion to form a unified multi-scale fusion representation, enhancing the ability to model target segments of different durations. S6: Frame-level feature local detail enhancement. Convolution operations are performed on windows of different scales for each frame to calculate the feature relationships within each local window. Local features at various scales are fused through a cross-attention mechanism to enhance the semantic expression of frame features and ensure that important local details can be captured in each video frame. S7: Semantic decomposition matching breaks down sentences into multiple semantic units, each corresponding to frame-level semantics. Word features are grouped into learnable semantic units through a projection layer, Transformer, and Slot Attention, ensuring that each unit covers specific content from the sentence that might appear in the frame-level video. Furthermore, a Semantic Diversity Loss is introduced to encourage diversity among the generated semantic units. S8: Similarity matching calculation, which calculates the matching degree between text and video, including the similarity between sentence vector and overall video representation (global matching) and the similarity between each semantic unit and all frames. Finally, the two are weighted and fused to form the final matching score. S9: Training optimization, using multiple loss functions in combination for training, including TripletRanking Loss, InfoNCE Loss, and Query Diverse Loss, to enhance the model's ability to express partial matching relationships and preserve semantic structure.

2. The method as described in claim 1, characterized in that, The frame-level sampling process in step S1 involves standardizing the number of video frames to no more than 128 frames. For videos exceeding this length, linear uniform sampling is used for dimensionality reduction.

3. The method as described in claim 1, characterized in that, The visual feature extraction module in step S2 includes: S21: Use ResNet152 as a 2D CNN to obtain static appearance features for each frame of the image. S22: Use I3D as a 3D CNN to obtain dynamic motion features of multi-frame sequences. Both are encoded separately, but the same embedding dimension is retained for subsequent fusion.

4. The method as described in claim 1, characterized in that, In the text feature encoding process described in step S3, the word vectors output by RoBERTa are modeled in context using Transformer and aggregated into sentence-level vectors through a learnable attention mechanism.

5. The method as described in claim 1, characterized in that, The attention mechanism of the Gaussian mixture modeling module described in step S4 constrains the inter-frame interaction weights through a Gaussian weight matrix. The output of the Gaussian attention module is: The Gaussian matrix is ​​defined as follows: Where i and j are frame indices, σ 2 The preset variance is used to control the range of attention spread.

6. The method as described in claim 1, characterized in that, The multi-scale fusion described in step S5 uses four types of Gaussian attention blocks, each corresponding to a different Gaussian variance (0.5, 1.0, 5.0, ∞), to adapt to semantic fragment information with different time spans. Finally, the blocks are aggregated and fused into a unified representation through average pooling.

7. The method as described in claim 1, characterized in that, The video frame-level feature refinement enhancement described in step S6 includes: S61: A multi-scale window attention mechanism that performs convolutional processing at different scales on each frame to extract video frame features from both local and global perspectives. The window attention mechanism can be described as follows: S62: Cross-attention mechanism, which cross-fuses the features of each local window through an attention mechanism to generate multi-scale enhanced features. S63: Residual connections and nonlinear mappings further enhance the semantic expressive power of frame features through residual connections and nonlinear fully connected layers. The final frame features are:

8. The method as described in claim 1, characterized in that, The semantic decomposition in step S7 includes: S71: Initialization. Randomly initialize n learnable vectors, called "slots". These slots are like empty containers, ready to be used to "hold" different semantic concepts. S72: Calculate the attention weights. For each slot, calculate its weights with respect to all word features W. f The similarity is calculated (usually using dot product or cosine similarity). Then, it is normalized using the Softmax function to obtain the "attention level" weight of each slot for each word. S73: Weighted aggregation update. For each slot, the features of all words are summed in a weighted manner based on the attention weights obtained in the previous step. After T iterations, the n slots are no longer random, and each captures a specific semantic subset of the sentence. These are the final semantic units.

9. The method as described in claim 1, characterized in that, The matching score calculation formula in step S8 is as follows: S(v,q)=α*S c (v,q)+(1-α)*S f (v,q)。

Images