A video-level grouping resampling method for visual text alignment in video understanding
Patent Information
- Application Number
- CN202611340676.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing video understanding models suffer from information redundancy and low learning efficiency in the visual text alignment process, making it difficult to effectively capture the temporal correlation between video frames. This results in poor performance when handling complex action sequences and dynamic scene understanding, failing to meet the needs of practical applications.
By employing a video-level group resampling method, and by initializing learnable query vectors and performing group cross-attention calculations, combined with positional encoding, lightweight processing of visual features and deep capture of temporal information are achieved, ensuring that the number of visual tokens is controllable and improving query learning efficiency and accuracy.
While maintaining a lightweight model, it significantly reduces computational resource consumption, improves the efficiency and accuracy of visual text alignment, enhances the ability to understand the temporal sequence of dynamic events and action sequences, and is suitable for efficient video understanding of multimodal models.
Smart Images

Figure CN123119352A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal large models, specifically relating to a video-level grouping resampling method for visual text alignment in video understanding. Background Technology
[0002] Video understanding, as a core supporting capability of multimodal intelligence, provides the technical foundation for key tasks such as action recognition and dynamic scene analysis, and drives the development of fields such as autonomous driving and intelligent security. However, due to the rich spatial information and temporal dynamic features contained in video data, as well as the complexity of processing long visual sequences, video understanding remains one of the core challenges in the field of computer vision.
[0003] In recent years, multimodal large-scale models have achieved breakthroughs in cross-modal information fusion, providing new technical pathways for video understanding. Existing methods typically sample a certain number of video frames, encode these frames using a visual encoder, perform visual-text alignment in a connector module, and concatenate the aligned visual features with textual features before inputting them into a language model to obtain a response. However, due to the inherent input length limitations of language models, previous methods often strictly limit the number of extracted frames through linear adjustments or introduce additional video encoders and complex memory storage mechanisms. These solutions either severely limit the acquisition of visual information, leading to the loss of a large amount of visual information, or result in a more complex model architecture, a surge in the number of parameters, significantly increasing the computational cost of training and inference, and reducing the feasibility of deploying the model on edge devices, making it difficult to meet the efficiency requirements of practical applications.
[0004] Furthermore, some methods focus on resampling techniques for single-frame videos, controlling the number of tokens by independently resampling features for each frame. While this approach can achieve a balance between information preservation and computational efficiency to some extent, it essentially decomposes a continuous video sequence into isolated static images for processing. It fails to break down the information barriers between individual frames and inherently ignores the temporal relationships between video frames. These temporal relationships are crucial for models to recognize complex action sequences and understand dynamic scenes. This makes these methods ineffective when handling tasks requiring temporal reasoning, struggling to accurately understand the deep logic of videos and failing to meet the demands for deep video understanding in practical applications. Summary of the Invention
[0005] To address the technical shortcomings of existing video understanding models in the process of visual text alignment, such as information redundancy and low learning efficiency, this invention discloses a video-level grouping resampling method for visual text alignment in video understanding. It aims to achieve a controllable reduction in the number of visual tokens while maintaining the lightweight nature of the modules through innovative grouping interaction design, while taking into account query learning efficiency and deep capture of inter-frame temporal information, thus providing technical support for efficient video understanding of multimodal models.
[0006] The resampling method based on video-level grouping has the following specific steps:
[0007] Step 1: Video frame sampling and visual feature extraction;
[0008] First, it receives input video (supporting common formats such as MP4, AVI, and MOV to adapt to video input needs in different scenarios). It selects the sampling mode according to the video duration and task requirements. Uniform frame sampling extracts images from the video at even intervals according to the preset number of sampling frames N; while FPS sampling extracts video frames according to the duration and frame rate of the video.
[0009] Then sample the video frame sequence Input to visual encoder In this context, the visual encoder can use models such as CLIP, SigLIP, and DinoV2. Through operations such as convolution and attention mechanisms, it extracts image features. The encoded visual feature sequence is... ,in This corresponds to the features of the i-th image.
[0010] Step 2: Initialize the learnable query vector and concatenate the visual feature sequences;
[0011] Step 201: First, initialize a fixed number of learnable queries. The query vector is defined by its shape as [Query_Num, Dim], where Query_Num is the preset number of queries, which can be flexibly adjusted according to computing resources and task accuracy requirements. Dim is the dimension of the query vector, and this dimension is strictly consistent with the dimension of subsequent text features to ensure direct fusion with text features. The query vector is generated through random initialization and continuously iteratively optimized during model training, enabling it to adaptively learn key information from video visual features and improve the accuracy of text alignment.
[0012] Step 202: Then, extract all the frame features obtained in Step 1. The frames are concatenated in temporal order to form a long sequence of global visual features. The shape of this visual sequence is [Per_Image_Token_Num * Frame_Num, Visual_Dim], where Per_Image_Token_Num is the number of visual tokens encoded in each image, Frame_Num is the number of sampled frames, and Visual_Dim is the dimension of the visual features. By concatenating them in temporal order, the temporal correlation information between video frames is fully preserved, avoiding the temporal fragmentation problem caused by single-frame processing.
[0013] Step 203: Finally, a linear transformation is applied to the long sequence of global visual features H to map the original dimension Visual_Dim of the visual features to the same dimension Dim as the query vector, thus obtaining the mapped visual features. Its shape is [Per_Image_Token_Num * Frame_Num, Dim]. This linear transformation layer learns the mapping relationship between visual features and text dimensions, achieving dimensional alignment between visual features and query vectors and text features, thus providing a prerequisite for cross-modal information interaction.
[0014] Step 3: Group the visual feature sequences and learnable queries;
[0015] Based on the set number of groups M, the learnable query Q is evenly divided into M query groups along the dimension of query quantity. The number of queries per group is Query_Num / M, ensuring that each group of queries has sufficient information learning capacity. Simultaneously, long sequences of global visual features are... Similarly, it is divided into M consecutive feature subsets. The number of frames corresponding to each feature subset is K = Frame_Num / M, and the shape of each feature subset is [Per_Image_Token_Num * Frame_Num / M, Dim], ensuring that each set contains continuous features of adjacent frames, laying the foundation for learning temporal information.
[0016] Through the grouping method described above, each feature subset contains video frame features in a continuous temporal sequence, providing a structural foundation for subsequent capture of inter-frame temporal dependencies. Simultaneously, each query group corresponds one-to-one with a feature subset, ensuring that each query group focuses solely on visual features within a specific temporal interval. This avoids ineffective interactions between queries and irrelevant features during global attention computation, improving the efficiency and accuracy of query learning.
[0017] Step 4: Group and interact with learnable queries and visual features;
[0018] This step is the core of the invention for capturing local visual information and inter-frame temporal correlation. The main process involves processing each query group... and the corresponding feature subset One-to-one cross-attention calculation is performed. As shown in Figure 2, this process is implemented through a multi-level stacked structure of cross-attention layers and feedforward networks (FFNs), and is the core step of this resampler in capturing local visual information and inter-frame temporal correlations. This module stacks layers according to the configured number to progressively enhance the expressive power of features. The specific execution process is as follows:
[0019] Step 401, Cross-attention calculation: Calculate the cross-attention of the grouped query groups. As a query in the cross-attention module, and simultaneously, feature subsets As the key and value, cross-attention is calculated using the following formula:
[0020]
[0021] Cross-attention computation calculates the similarity between the query and the key, enabling the query to focus on and extract key information from the feature subset, thus allowing the query group to adaptively learn the core visual features of the corresponding time interval.
[0022] Step 402, Residual Connection and Feedforward Network Transformation: Perform residual connection on the output of cross attention to alleviate the gradient vanishing problem in deep network training. Then, input the result of the residual connection into the feedforward network (FFN) for feature transformation. Similarly, superimpose residual connection on the transformed result. Through the above operations, complete the interactive calculation of single-round attention + feedforward network.
[0023] Step 403: Repeat the above calculation process of cross-attention and feedforward network, gradually enhancing the feature representation ability of the query group through multi-layer interaction, and finally obtaining the optimized local query group. .
[0024] Step 5: Concatenate the learnable queries in chronological order and add location codes;
[0025] First, the M optimized local query groups The corresponding temporal order is then sequentially assembled along the dimension of query quantity to form the final visual query sequence. Its shape is restored to [Query_Num, Dim], which is perfectly aligned with the word embedding dimension of the language model, laying the foundation for dimension matching for subsequent multimodal fusion.
[0026] Then, the assembled global visual query sequence Add position encoding, which can be sine or cosine position encoding or learnable position encoding, to distinguish the temporal and spatial positions of feature vectors in a sequence.
[0027] The query sequence with added positional encoding can be directly concatenated with the text embedding to form a feature sequence for visual-text fusion, which is then input into a language model for multimodal fusion and understanding. This fused feature sequence can simultaneously carry the visual key information of the video, the inter-frame temporal logic, and the semantic information of the text, and is ultimately input into a language model for multimodal fusion and video understanding tasks (such as action recognition, dynamic scene parsing, and video question answering).
[0028] The advantages of this invention are:
[0029] (1) A video-level grouping resampling method for visual text alignment in video understanding, which is based on setting a fixed number of learnable queries and image feature sequences for cross-attention, which significantly reduces the number of visual tokens and reduces computational resource consumption.
[0030] (2) A video-level grouping resampling method for visual text alignment in video understanding avoids the query redundancy defect in traditional image-level resampling by uniformly grouping the feature sequence and query and performing corresponding cross-attention calculation, thereby improving query learning efficiency and information capture accuracy and ensuring the model’s advantage in fine-grained visual detail understanding.
[0031] (3) A video-level grouping resampling method for visual text alignment in video understanding, which relies on grouping learning of continuous frame features, can efficiently capture the temporal dependencies between video frames and enhance the model's ability to understand the temporal sequence of dynamic events and action sequences. Attached Figure Description
[0032] Figure 1 This is a flowchart of a video-level grouping resampling method for visual text alignment in video understanding according to the present invention;
[0033] Figure 2 This is an example diagram illustrating the architecture of a video-level grouping resampling method for visual text alignment in video understanding, as proposed in this invention. Detailed Implementation
[0034] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.
[0035] This invention discloses a video-level grouping resampling method for visual-text alignment in video understanding. Applied to multimodal models, it employs an innovative design that interactively learns the correspondence between visual feature sequences and learnable query groups. This reduces attention redundancy and enhances temporal information capture while controlling the number of visual tokens, significantly contributing to efficient visual-text alignment. This method is adaptable to different sampling methods and video scenarios of varying durations, and can be widely applied in fields such as action recognition, dynamic scene analysis, and autonomous driving perception.
[0036] The technical solution of this invention is implemented based on standardized deep learning training and inference frameworks (such as PyTorch), and the hardware environment supports NVIDIA GPUs (including A100, 3090, 4060 Ti, etc.). Figure 1 As shown, the specific implementation steps are as follows:
[0037] Step 1: Video frame sampling and visual feature extraction;
[0038] First, it receives input video (supporting common formats such as MP4, AVI, and MOV to adapt to video input needs in different scenarios). It selects the sampling mode according to the video duration and task requirements. Uniform frame sampling extracts images from the video at even intervals according to the preset number of sampling frames N; while FPS sampling extracts video frames according to the duration and frame rate of the video.
[0039] Then sample the video frame sequence Input to visual encoder In this context, the visual encoder can use models such as CLIP, SigLIP, and DinoV2. Through operations such as convolution and attention mechanisms, it extracts image features. The encoded visual feature sequence is... ,in This corresponds to the features of the i-th image.
[0040] Step 2: Initialize the learnable query vector and concatenate the visual feature sequences;
[0041] Step 201: First, initialize a fixed number of learnable queries. The query vector is defined by its shape as [Query_Num, Dim], where Query_Num is the preset number of queries, which can be flexibly adjusted according to computing resources and task accuracy requirements. Dim is the dimension of the query vector, and this dimension is strictly consistent with the dimension of subsequent text features to ensure direct fusion with text features. The query vector is generated through random initialization and continuously iteratively optimized during model training, enabling it to adaptively learn key information from video visual features and improve the accuracy of text alignment.
[0042] Step 202: Then, extract all the frame features obtained in Step 1. The frames are concatenated in temporal order to form a long sequence of global visual features. The shape of this visual sequence is [Per_Image_Token_Num * Frame_Num, Visual_Dim], where Per_Image_Token_Num is the number of visual tokens encoded in each image, Frame_Num is the number of sampled frames, and Visual_Dim is the dimension of the visual features. By concatenating them in temporal order, the temporal correlation information between video frames is fully preserved, avoiding the temporal fragmentation problem caused by single-frame processing.
[0043] Step 203: Finally, a linear transformation is applied to the long sequence of global visual features H to map the original dimension Visual_Dim of the visual features to the same dimension Dim as the query vector, thus obtaining the mapped visual features. Its shape is [Per_Image_Token_Num * Frame_Num, Dim]. This linear transformation layer learns the mapping relationship between visual features and text dimensions, achieving dimensional alignment between visual features and query vectors and text features, thus providing a prerequisite for cross-modal information interaction.
[0044] Step 3: Group the visual feature sequences and learnable queries;
[0045] Based on the set number of groups M, the learnable query Q is evenly divided into M query groups along the dimension of query quantity. The number of queries per group is Query_Num / M, ensuring that each group of queries has sufficient information learning capacity. Simultaneously, long sequences of global visual features are... Similarly, it is divided into M consecutive feature subsets. The number of frames corresponding to each feature subset is K = Frame_Num / M, and the shape of each feature subset is [Per_Image_Token_Num * Frame_Num / M, Dim], ensuring that each set contains continuous features of adjacent frames, laying the foundation for learning temporal information.
[0046] Through the grouping method described above, each feature subset contains video frame features in a continuous temporal sequence, providing a structural foundation for subsequent capture of inter-frame temporal dependencies. Simultaneously, each query group corresponds one-to-one with a feature subset, ensuring that each query group focuses solely on visual features within a specific temporal interval. This avoids ineffective interactions between queries and irrelevant features during global attention computation, improving the efficiency and accuracy of query learning.
[0047] Step 4: Group and interact with learnable queries and visual features;
[0048] This step is the core of the invention for capturing local visual information and inter-frame temporal correlation. The main process involves processing each query group... and the corresponding feature subset One-to-one cross-attention calculation is performed. As shown in Figure 2, this process is implemented through a multi-level stacked structure of cross-attention layers and feedforward networks (FFNs), and is the core step of this resampler in capturing local visual information and inter-frame temporal correlations. This module stacks layers according to the configured number to progressively enhance the expressive power of features. The specific execution process is as follows:
[0049] Step 401, Cross-attention calculation: Calculate the cross-attention of the grouped query groups. As a query in the cross-attention module, and simultaneously, feature subsets As the key and value, cross-attention is calculated using the following formula:
[0050]
[0051] Cross-attention computation calculates the similarity between the query and the key, enabling the query to focus on and extract key information from the feature subset, thus allowing the query group to adaptively learn the core visual features of the corresponding time interval.
[0052] Step 402, Residual Connection and Feedforward Network Transformation: Perform residual connection on the output of cross attention to alleviate the gradient vanishing problem in deep network training. Then, input the result of the residual connection into the feedforward network (FFN) for feature transformation. Similarly, superimpose residual connection on the transformed result. Through the above operations, complete the interactive calculation of single-round attention + feedforward network.
[0053] Step 403: Repeat the above calculation process of cross-attention and feedforward network, gradually enhancing the feature representation ability of the query group through multi-layer interaction, and finally obtaining the optimized local query group. .
[0054] Specifically, the components and functions of a feedforward neural network are as follows:
[0055] 1. Layer Normalization Layer: Using the feature dimension Dim as the normalization dimension, the layer normalizes the input feature vector, which can stabilize the data distribution, alleviate the gradient vanishing problem, and accelerate the training and convergence of the model.
[0056] 2. Linear Layer: As a dimension-upgrading layer, it takes a input dimension of dim and an output dimension of dim × mult (where mult is the dimension scaling factor, with a default value of 4), and does not set a bias (bias=False). The purpose of this layer is to map the feature vectors to a higher-dimensional space, providing a richer feature space for subsequent nonlinear transformations.
[0057] 3. GELU Activation Function Layer: The Gaussian Error Linear Unit (GELU) is used as the non-linear activation function. Compared with traditional activation functions such as ReLU, GELU has smooth non-linear characteristics, which can better preserve the fine-grained information of features, avoid the problem of hard truncation of activation values, and improve the flexibility of feature transformation.
[0058] 4. Linear Layer: As a dimensionality reduction layer, the input dimension is dim × mult, and the output dimension is restored to dim. No bias is set (bias=False). This layer maps the high-dimensional feature vector back to the original dimension, achieving a balance between non-linear transformation of features and dimensionality restoration, ultimately outputting an enhanced feature vector with the same dimension as the input.
[0059] Step 5: Concatenate the learnable queries in chronological order and add location codes;
[0060] First, the M optimized local query groups The corresponding temporal order is then sequentially assembled along the dimension of query quantity to form the final visual query sequence. Its shape is restored to [Query_Num, Dim], which is perfectly aligned with the word embedding dimension of the language model, laying the foundation for dimension matching for subsequent multimodal fusion.
[0061] Then, the assembled global visual query sequence Add position encoding, which can be sine or cosine position encoding or learnable position encoding, to distinguish the temporal and spatial positions of feature vectors in a sequence.
[0062] The query sequence with added positional encoding can be directly concatenated with the text embedding to form a feature sequence for visual-text fusion, which is then input into a language model for multimodal fusion and understanding. This fused feature sequence can simultaneously carry the visual key information of the video, the inter-frame temporal logic, and the semantic information of the text, and is ultimately input into a language model for multimodal fusion and video understanding tasks (such as action recognition, dynamic scene parsing, and video question answering).
[0063] The embodiments of the video-level block resampler described in this specification can be implemented using tangible, concrete computer software or firmware, computer hardware (including the structures disclosed herein and their equivalents), or a combination of one or more of these. The embodiments of the video-level block resampler described in this specification can be implemented as one or more computer programs. A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope of the conclusions of the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A video-level grouping resampling method for visual text alignment in video understanding, characterized in that, The specific steps are as follows: First, video frame sampling and visual feature extraction are performed: The input video is received, and N video frame sequences are obtained by sampling according to a preset pattern. After preprocessing, these sequences are input into a visual encoder, which outputs a frame feature sequence. ; Then, learnable query initialization and feature concatenation are performed: a fixed number of learnable queries are initialized. The frame feature sequences are concatenated into a long global visual feature sequence, and the mapped visual feature vector is obtained by unifying the dimensions through linear transformation. . Next, grouping and interactive learning are performed: the mapped visual feature vectors are... Divide into M consecutive feature subsets At the same time, the learnable query Divided into M query groups For each query group and the corresponding feature subset One-to-one cross-attention calculation is performed, combined with a feedforward network for nonlinear feature transformation, and the optimized local query group is obtained through multi-layer stacking iteration. . Finally, the queries are concatenated and location codes are added: the M local query groups are... By concatenating the data along the query quantity dimension, a global visual query sequence is formed. After adding positional encoding, the final query sequence is obtained. This is then concatenated with the text embedding sequence and input into the language model to complete the multimodal task.
2. The video-level grouping resampling method for visual text alignment in video understanding as described in claim 1, characterized in that, The visual encoder uses models including but not limited to CLIP, SigLIP, and Dinov2 to extract image features.
3. The video-level grouping resampling method for visual text alignment in video understanding as described in claim 1, characterized in that, The number of groups M for grouping the query and visual feature sequences is less than the number of sampling frames N. The number of frames corresponding to each feature subset is K=N / M, and the value of K is usually an integer greater than or equal to 2.
4. The video-level grouping resampling method for visual text alignment in video understanding as described in claim 1, characterized in that, The group interactive learning described in claim 1 specifically includes: 1) Cross-attention calculation: based on learnable queries As a query, feature subset As the key and value, cross-attention is calculated using the following formula: 2) Residual Connections and Feature Transformation: Residual connections are performed on the output of the cross-attention process, which is then input into the feedforward network (FFN) for feature transformation. Similarly, residual connections are superimposed on the transformed results. Through these operations, the computation of a single round of attention + feedforward network is completed. 3) Multi-layer iteration: Repeat the above computational process of cross-attention and feedforward network, and gradually enhance the feature representation ability of the query group through multi-layer interaction, finally obtaining the optimized local query group. .
5. A video-level grouping resampling method for visual text alignment in video understanding as described in claim 1, characterized in that, The feedforward network adopts a sequential structure of layer normalization + linear dimensionality increase + GELU activation + linear dimensionality reduction, specifically including: 1) Layer normalization layer: Performs layer normalization on the input features to stabilize the data distribution; 2) First linear layer: The input dimension is Dim, and the output dimension is Dim×mult, where mult is the dimension scaling factor, the default value is 4, and no bias is set; 3) GELU activation function layer: achieves smooth nonlinear transformation of features; 4) Second linear layer: The input dimension is Dim×mult, and the output dimension is restored to Dim. No bias is set.
6. The video-level grouping resampling method for visual text alignment in video understanding as described in claim 1, characterized in that, The absolute position encoding described in claim 1 includes sine and cosine position encoding or learnable position encoding, which are added element-by-element to the global visual query sequence to obtain the final query sequence, and then input into the language model for multimodal fusion and understanding.
