Multi-modal video understanding method based on adaptive optimization

Abstract

The invention discloses a multi-modal video understanding method based on adaptive optimization, and belongs to the field of multi-modal semantic modeling. Aiming at the problem that modal weight cannot be dynamically adjusted by static fusion indicated in the background technology, the content of the invention provides an AOSM module for the first time: a frame-level attention weight is generated on line through a learnable temperature parameter tau, and real-time quantification of text-frame semantic relevancy is realized. The AOSM provides two processing modes: residual enhancement maintains all frames and strengthens key frames; according to the Top-K hard screening, redundant frames are pruned on the premise that time sequence continuity is kept, and the two modes can be switched according to tasks. Semantic consistency constraints are introduced into the module, and vision-text features are aligned through a unified embedding space. End-to-end training of the whole frame does not need extra supervision signals and parameter quantity lt; and 0.3 M. An HMDB-51 experiment result shows that the accuracy rate of Top-1 reaches 66.4%, and the effectiveness and deployment friendliness of a self-adaptive optimization strategy are verified.

Description

Technical Field

[0001] This invention belongs to the field of multimodal video understanding technology based on adaptive optimization. Specifically, it relates to a method for video action recognition through adaptive weight allocation and semantic modeling within a vision-language pre-training framework, achieving high-precision, low-redundancy semantic enhancement and keyframe selection fusion. Background Technology

[0002] 1. The Rise of Multimodal Video Understanding: In recent years, visual content, represented by short videos, live streaming, and wearable devices, has experienced explosive growth. Traditional intelligent analysis based solely on "image viewing" or "sound listening" is no longer sufficient to meet the high-precision requirements of complex scenarios. Researchers have begun to incorporate visual, text, audio, and even sensor signals into a unified framework, forming the Multimodal Video Understanding (MVU) task. Its core challenge is how to align heterogeneous signals in the spatiotemporal dimensions and utilize complementary information to suppress unimodal noise, thereby improving the accuracy of action recognition, temporal localization, and event description.

[0003] 2. Visual-Language Pre-training Provides a New Foundation: With the emergence of large-scale visual-language pre-trained (VLP) models such as CLIP, ALIGN, and Florence, image-text contrastive learning has demonstrated the transferability of "semantic alignment." Transferring VLP to the video domain allows for the inheritance of powerful static visual representations while leveraging open-class supervision through open-text vocabulary, alleviating the problems of closed categories and expensive annotation in traditional video datasets. However, the high redundancy of video sequences, short action durations, and concise, abstract text labels make the crude approach of "directly averaging frame features + text matching" insufficient for capturing crucial action moments, resulting in inadequate fine-grained discriminative capabilities.

[0004] 3. The conflict between frame redundancy and computational efficiency: Existing methods generally process 8–32 frames obtained from uniform sampling with equal weights, of which background and transition frames often account for more than 60%. While average pooling is simple, it weakens motion peaks; max pooling retains only a single frame, losing motion continuity. Some works introduce temporal Transformers or 3D convolutions, but the computational cost increases linearly or even quadratically with the number of frames, making it unsuitable for edge deployment. Finding a balance between accuracy and computation has become a bottleneck for the practical application of multimodal action recognition.

[0005] 4. The granularity gap in cross-modal interaction: Video frame features are typically spatiotemporal cubes (T×H×W×D), while category text contains only a few words (L≈5~10). The two differ significantly in sequence length, feature scale, and information density. Directly performing dot products or cross-attention can easily introduce "noise matching"—that is, text and background frames generate false high responses, misleading subsequent classification. Works such as BIKE and X-CLIP have attempted to alleviate this problem through bidirectional knowledge exploration, but they still employ a static paradigm of "full-frame fusion followed by classification," lacking explicit and learnable control over "which frame is most worth watching."

[0006] 5. Opportunities and Limitations of Attention Mechanisms in Modality Fusion: Attention mechanisms, with their dynamic weight allocation advantages, have achieved significant results in tasks such as image-text retrieval and video question answering. However, directly applying them to video action recognition still faces the following technical challenges and limitations:

[0007] (1) Video frame redundancy and information dilution problem: Video sequences naturally contain a lot of temporal redundancy information, such as static background, repeated actions or transition frames. Existing methods usually perform equal weight processing on all sampled frames (such as average pooling) or use complex global temporal modeling (such as 3D convolution, temporal Transformer), which causes the signal of key action frames to be diluted by a large number of non-key frames, making it difficult for the model to focus on the decisive moment of action and affecting the recognition accuracy.

[0008] (2) Difficulty in fine-grained alignment between text and visual modalities: In video-text cross-modal tasks, short category-level text descriptions (such as "playing basketball") need to be aligned with video sequences that are several seconds long and contain complex spatiotemporal information. Existing cross-modal attention mechanisms often compute between global video features and text features, lacking explicit, fine-grained guidance on "which part (frames) of the video the text description should be most relevant to", which can easily lead to fuzzy or incorrect matching, especially in scenes with complex backgrounds or multiple actions.

[0009] (3) The contradiction between computational efficiency and model complexity: In order to capture long-term temporal dependencies, many advanced models adopt multi-layer Transformer or 3D convolutional networks. Their computational complexity increases linearly or quadratically with the number of frames, which places a heavy burden on model training and deployment, especially on resource-constrained edge devices. How to reduce computational overhead while ensuring performance is an urgent problem to be solved.

[0010] (4) Lack of interpretable and learnable frame selection mechanisms: Although some studies have attempted to select keyframes using rule-based (e.g., random dropping) or heuristic (based on motion energy) methods, these methods are not end-to-end learnable and cannot adaptively optimize according to specific recognition tasks and text semantics. Other attention-based methods can learn weights, but their attention is usually applied in the feature fusion stage rather than in the earlier frame selection stage, thus having limited effect on reducing computational redundancy.

[0011] However, existing methods mostly rely on static fusion strategies and lack adaptive optimization mechanisms that can dynamically adjust modal contributions based on video content, making it difficult to maintain a balance between semantic consistency and computational efficiency in complex scenarios. To address these issues, this invention proposes a novel text-guided adaptive frame selection and enhancement method. The core of this method lies in inserting a lightweight, learnable attention selection module after video feature encoding and before complex cross-modal fusion. This module uses task-related text semantics as a query to dynamically calculate the importance weight of each frame, thereby enhancing key frames or suppressing redundant frames. This method optimizes the feature quality input to subsequent complex modules from the source, providing a new technical path to solve frame redundancy, improve semantic alignment accuracy, and reduce computational costs to some extent. Summary of the Invention

[0012] The AOSM (Adaptive Optimization and Semantic Modeling) module described in this invention achieves adaptive weight allocation and semantic enhancement of multimodal features through learnable parameters. The aim is to overcome the shortcomings of existing technologies and provide a text-guided adaptive frame selection and enhancement method for video action recognition. This method aims to improve the discriminative power of video representation and enhance the model's ability to focus on core action content by introducing a keyframe selection and enhancement module that is linked to text semantics and can be trained end-to-end. It also provides a simple and easily integrated new approach for multimodal video understanding.

[0013] A text-guided adaptive frame selection and enhancement method for video action recognition, characterized by the following steps:

[0014] Step 1: Multimodal Feature Extraction

[0015] Input a batch of video data and its corresponding category text labels. First, use a visual encoder (such as the CLIP visual encoder based on Vision Transformer) to extract features from T uniformly sampled frames of each video, obtaining frame-level visual feature tensors. Where B is the batch size, T is the number of frames, and D is the feature dimension. Simultaneously, a text encoder (such as the CLIP text encoder) is used to encode the text tags to obtain the text semantic feature vector. .

[0016] Step 2: Text-guided frame attention generation

[0017] The visual features V and text features T obtained in step one are input into an adaptive frame selection module. This module first normalizes the input features, then calculates the similarity between the text features and the visual features of each frame, and finally generates an attention weight distribution that reflects the semantic importance of each frame relative to the text through a learnable temperature-adjustable Softmax function. ,and The specific calculation formula is as follows:

[0018]

[0019] Where L2-Norm represents L2 normalization along the feature dimension, making the vector magnitude 1.

[0020] Calculate the similarity matrix using the following formula:

[0021]

[0022] in , which is the semantic similarity matrix between each frame and the text within the batch.

[0023]

[0024] Here, τ is a learnable scalar temperature parameter used to control the sharpness of the attention distribution. The initial value can be set to 0.07 to control the sharpness of the attention distribution.

[0025] Step 3: Attention-based frame feature processing

[0026] Based on the attention weights A generated in step two, the original frame-level features V are processed to generate enhanced video features. This invention provides two processing modes:

[0027] Mode 1: Soft Enhancement. This mode retains all frames but recalibrates the feature amplitudes according to attention weights, enhancing important frames and weakening less important frames. The specific operation is as follows:

[0028]

[0029] Where ⊗ represents element-wise multiplication broadcast along the feature dimension D; 1+A is the residual enhancement coefficient. This mode keeps the time series length unchanged and is suitable for modules that require fine-grained time series modeling later.

[0030] Mode 2: Hard Selection. In this mode, the most important K frames (K < T) are selected according to the attention weights to form a more compact video representation. To maintain the relative order of the temporal information, the frame indices idx need to be re-sorted in chronological order after the selection. The specific operations are as follows:

[0031]

[0032] K is the preset number of frames to be retained, with a value range of 4 to 12; idx is the original index of the K frames with the highest weights.

[0033]

[0034] Re-sort in chronological order to maintain the video temporal continuity.

[0035]

[0036] Gather means extracting the corresponding frame features according to the indices; the output dimension K is the selected number of frames, and D is the feature dimension.

[0037] This mode can directly reduce the computational complexity of subsequent processing.

[0038] Step 4: Cross-modal Fusion and Classification

[0039] Input the processed video features and the text feature T into the subsequent cross-modal fusion layer (such as a Transformer-based interaction module or a video classification head) for in-depth inter-modal information interaction, and finally generate the prediction scores (logits) of the action categories.

[0040] Step 5: Model Training and Optimization

[0041] Use the standard loss function for video action recognition tasks (such as cross-entropy loss) to perform end-to-end training on the model. During the training process, the learnable temperature parameter τ in the adaptive frame selection module and other model parameters are optimized together by the gradient descent method.

[0042] The beneficial effects of this invention are as follows:

[0043] Introduce a text semantic guidance mechanism: Innovatively use task-related text descriptions as prior information to guide the importance evaluation of video frames, making the frame selection process have clear semantic directivity, which helps the model focus on the visual content most relevant to the action category.

[0044] Achieving adaptive and learnable frame selection: Through a learnable attention mechanism and temperature parameters, the model can dynamically adjust the frame selection strategy according to different datasets and tasks, avoiding the limitations of manually designed rules and improving the generalization ability and flexibility of the method.

[0045] It offers flexible processing modes: soft enhancement and hard selection modes are designed for different application scenarios that prioritize accuracy and efficiency, respectively, enabling the invention to be flexibly adapted to various video understanding model architectures and deployment requirements.

[0046] Lightweight and easy to integrate: The proposed adaptive frame selection module has a simple structure and very few parameters. It can be easily embedded into existing video understanding frameworks based on vision-language pre-trained models (such as BIKE, X-CLIP, etc.) as a plug-and-play component without significantly modifying the original model structure.

[0047] Enhanced feature discriminativeness: By enhancing keyframes or removing redundant frames, the video features output to subsequent modules become more discriminative, which theoretically helps improve the model's ability to distinguish difficult examples and enhances the model's robustness.

[0048] Compared to traditional fixed-weighted fusion, the adaptive optimization strategy proposed in this invention enables the model to have stronger semantic focusing ability and cross-modal generalization performance in complex video scenarios. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0050] Figure 2 A schematic diagram of the internal structure and two processing modes of the Adaptive Frame Selection Module (AOSM);

[0051] Figure 3 A visual example of guiding attention weights for text;

[0052] Figure 4 This is a schematic diagram of the frame selection and reordering process in hard selection mode; Specific implementation methods

[0053] Example 1: Overall Structure and Data Flow

[0054] Step 1: Input Data and Feature Extraction

[0055] like Figure 1 As shown, this implementation first reads the raw input video from a predefined video dataset. Data samples are obtained from the data loader in batches, each sample containing:

[0056] Video frame sequences (images): Video segments composed of consecutive frames;

[0057] Text descriptions: Semantic text corresponding to the video (e.g., category labels or natural language descriptions);

[0058] Category index list_id: The category number to which the video belongs;

[0059] The ground_truth label is the real label required for supervised training.

[0060] Video frames are input into a Visual Transformer (ViT) model to obtain image embedding features for each frame; text is input into a Text Transformer model to obtain semantic embedding features.

[0061] The output includes:

[0062]

[0063] Where 𝑉 is the image embedding feature sequence, 𝑇 is the text embedding feature, 𝐵 represents the batch size, 𝑇 represents the number of video frames, and 𝐷 represents the embedding dimension.

[0064] This step provides the basic feature input for subsequent modality fusion. The ViT model is used to extract global visual features from the spatial dimension, and the text Transformer is used to extract contextual semantic vectors. The outputs of the two are aligned in the semantic space.

[0065] Step 2: Initialization and Feature Input of the AOSM Module

[0066] like Figure 2 As shown, the visual and textual features output from Step 1 are input into the AOSM (Adaptive Optimized Semantic-guided Module). This module is responsible for calculating the weight distribution for each frame based on the semantic information of the text. First, the input features are normalized:

[0067]

[0068] Then, frame-level semantic similarity is calculated:

[0069]

[0070] Where S represents the semantic similarity matrix; b is the batch index; t is the frame index; and d is the feature dimension index.

[0071] Then, the weight smoothness is controlled and normalized by the temperature parameter τ:

[0072]

[0073] Where A is the frame-level attention weight matrix; τ is the temperature parameter; and ε is a perturbation constant to prevent the denominator from being zero. The resulting attention matrix A is used to represent the semantic importance of each frame.

[0074] This step guides the visual frame features through textual semantics, enabling the model to adaptively allocate frame-level weights under different semantic conditions. The parameter τ can automatically adjust the concentration of attention distribution according to different tasks.

[0075] Step 3: Feature Enhancement and Frame Weighted Fusion

[0076] like Figure 3 As shown, the AOSM module provides two enhancement modes: Weighted (soft enhancement) and Top-K (keyframe selection).

[0077] Step 3.1 Weighted Mode (Residual Weighting Enhancement)

[0078] In Weighted mode, the AOSM module performs weighted enhancement on all frames, using the following enhancement formula:

[0079]

[0080] Output the enhanced feature sequence 𝑉′ while maintaining consistency in the time dimension.

[0081] Step 3.2 Top-K Mode (Keyframe Selection)

[0082] In the Top-K model, only the frame with the highest semantic score is selected for subsequent temporal modeling. The selection formula is as follows:

[0083]

[0084]

[0085] Here, `topk()` returns the indices of the K frames with the highest attention weights in the original video sequence, and `argsort` sorts the frames according to their chronological order in the original time dimension, outputting the features. This is used to reduce computational load and enhance semantic focus.

[0086] This step achieves semantically driven feature selection through attention weights. The Weighted mode preserves temporal continuity, while the Top-K mode enhances semantic focus. The two modes can be dynamically switched according to task requirements.

[0087] Step 4: Temporal Fusion and Semantic Matching

[0088] like Figure 4 As shown, the enhanced video features are input into the temporal Transformer module. The module calculates inter-frame dependencies through a multi-head attention mechanism and extracts the global temporal features of the video, 𝐹.

[0089] The calculation process is as follows:

[0090]

[0091] in, This is a learnable projection matrix used to generate the attention head.

[0092]

[0093] in, The feature dimension is used to scale the dot product and prevent gradient saturation.

[0094] After outputting the temporal features, a video-level representation is obtained through global pooling. and text embedding Calculate the semantic matching score:

[0095]

[0096] s is the video-text cosine similarity, with a value range of [−1,1], which is used for subsequent classification or retrieval.

[0097] This step establishes cross-modal semantic alignment. The temporal Transformer captures dynamic inter-frame dependencies, ensuring the continuity of enhanced frame features in the temporal dimension while maintaining text-video semantic consistency.

[0098] Step 5: Loss Function and Model Update

[0099] The model uses the Dual InfoNCE contrastive loss function, defined as:

[0100]

[0101] Where λ is the temperature scaling factor, which is taken as 0.07 in the experiment; Let represent the similarity between the _i video sample and the _i text sample, and let _i_ be the scaling temperature.

[0102] This step implements cross-modal bidirectional constraints to ensure consistency in video-text matching, while improving frame-level semantic resolution capabilities through an adaptive attention mechanism.

[0103] Step 6: Output and Deployment

[0104] After training, the model only needs to input a sequence of video frames during the inference phase to automatically calculate the frame-level attention distribution and output the enhanced feature 𝑉′. The output includes:

[0105] Frame attention heatmaps are used for visualization and analysis.

[0106] Video semantic features Used for classification or retrieval;

[0107] Matching scores are used for cross-modal alignment.

[0108] This process can run on standard GPU platforms, supports dynamic frame selection and semantic visualization, and can be extended to tasks such as video retrieval, action recognition, and video generation.

[0109] This embodiment fully demonstrates the input, processing, and output flow of AOSM-BIKE. Each step focuses on semantic relevance, combining a visual Transformer with a temporal modeling structure to achieve semantic enhancement of video features. Figures 1 to 4 The process and structure are shown. This method significantly improves frame-level semantic consistency and feature representation ability while maintaining the simplicity of the model structure.

[0110] Specific Implementation Method Two: AOSM Module Algorithm and Parameter Adjustment Process

[0111] like Figure 2 and Figure 3 As shown, this embodiment details the internal algorithm structure and parameter adjustment method of AOSM (Adaptive Optimized Semantic-guided Module). The design goal of this module is to dynamically adjust the weight distribution of video frame features based on textual semantic information, thereby achieving frame-level feature optimization before temporal feature fusion.

[0112] Step 1: Module Input and Normalization

[0113] The input consists of visual feature 𝑉 and text feature 𝑇 obtained from Implementation Method 1, with the following dimensions:

[0114]

[0115] Where 𝐵 is the batch size, 𝑇 is the number of video frames, and 𝐷 is the embedding dimension.

[0116] The module first performs L2 normalization on both visual and textual features to ensure consistent scale during subsequent similarity calculations. The normalization process is as follows:

[0117]

[0118] Here, ε is a perturbation term to prevent the denominator from being zero.

[0119] This step ensures that features of different modalities are on the same distribution scale, which helps stabilize subsequent similarity calculations and attention normalization processes.

[0120] Step 2: Semantic Relevance Calculation

[0121] The module obtains the semantic relevance matrix K by calculating the cosine similarity between text features and visual features of each frame:

[0122]

[0123] Then, by adjusting the smoothness of the correlation distribution using the temperature parameter τ, an attention distribution A is generated.

[0124]

[0125] Output Tensor This represents the frame-level semantic weight for each sample.

[0126] This step enables explicit modeling of frame features at the semantic level. The temperature parameter τ determines the degree of focus of the attention distribution; a small τ value concentrates the weights on a few frames, while a large τ value makes the weights more even.

[0127] Step 3: Weighted Mode (Residual Weighted Enhancement)

[0128] like Figure 3 As shown, when set to Weighted mode, the features of all frames will be residual-enhanced based on semantic weights. The enhancement formula is as follows:

[0129]

[0130] The enhanced output V′ has the same shape as the input, and its dimensions remain the same. .

[0131] This model maintains the integrity of the temporal dimension without discarding any frame information. Through an additive residual enhancement mechanism, the model can strengthen semantic keyframe features without disrupting temporal continuity.

[0132] Step 4: Top-K mode (semantic keyframe selection)

[0133] like Figure 4 As shown, when the model enables Top-K mode, only the frame with the highest semantic score is retained. The implementation process includes:

[0134] Calculate attention score ;

[0135] Select the first K largest weight frame indices ;

[0136] Reorder these indexes in chronological order;

[0137] Extract the features of the corresponding frames to construct a new sequence 𝑉′.

[0138] The formula is as follows:

[0139]

[0140]

[0141] The Top-K pattern reduces the length of the temporal input, allowing the model to focus on the most semantically significant frames, making it suitable for real-time inference or scenarios where device computing power is limited.

[0142] Step 5: Initialization and adjustment of temperature parameter 𝜏

[0143] In the AOSM module, the temperature parameter φ is a learnable variable that automatically adjusts during training. Initialization settings:

[0144] During training, it is constrained to be positive using the Softplus function:

[0145]

[0146] The constrained positive temperature value. To ensure that the training process is minor .

[0147] This design ensures that τ is smooth, differentiable, and numerically stable throughout the training process. The adjustment mechanism allows the model to adaptively adjust its focus based on data complexity.

[0148] Step 6: Output and Post-processing

[0149] The AOSM module outputs enhanced video features 𝑉′ and frame-level attention matrix 𝐴.

[0150] The output will be directly input into the temporal Transformer module for subsequent inter-frame feature modeling.

[0151] This module outputs two types of information: enhanced video features (for model use) and attention weights (which can be used for visualization or feature interpretation). As a standalone, pluggable unit, the AOSM module can be flexibly integrated into different types of video-text models.

[0152] This implementation details the input format, normalization method, semantic weight calculation, two enhancement modes, and parameter adjustment process of the AOSM module. The entire process requires no additional supervision signals; all calculations can be optimized through end-to-end backpropagation. Figure 2 and Figure 3 As can be seen, this module has a simple structure and clear logic, and can effectively achieve semantically guided frame-level feature optimization, providing high-quality input for subsequent temporal modeling.

[0153] Specific Implementation Method 3: Temporal Fusion and Cross-Modal Feature Process

[0154] like Figure 3 and Figure 4 As shown, this implementation illustrates how to perform temporal fusion and cross-modal semantic alignment after completing frame-level feature enhancement of the AOSM module. The core of this step lies in using the Temporal Transformer model to model inter-frame dependencies in the temporal dimension, and achieving consistent matching between text semantics and video dynamics through cross-modal feature fusion.

[0155] Step 1: Input Feature and Location Encoding

[0156] The enhanced frame-level features output by the AOSM module are represented as follows:

[0157]

[0158] Where 𝐵 represents the batch size, 𝑇 represents the number of frames, and 𝐷 represents the feature dimension. To enable the Transformer model to recognize temporal order, frame positional embedding is added to the features of each frame, calculated as follows:

[0159]

[0160]

[0161] in This represents frame positional embedding, which is the positional information at each time step, and is usually obtained by looking up the embedding matrix. This represents the frame features after adding position encoding, which is achieved by modifying the original frame features. With position encoding The purpose of adding them together is to provide time sequence information for the model.

[0162] This step ensures that the temporal order of the input sequence is preserved, enabling the model to distinguish frame features at different time points and avoid semantic confusion caused by disordered input.

[0163] Step 2: Multi-layer attention modeling of temporal Transformer

[0164] like Figure 3 As shown, the enhanced frame feature sequence The input is fed into a multi-layer residual attention network. Each Transformer layer consists of a multi-head attention layer and a feedforward network. The calculation process is as follows:

[0165]

[0166]

[0167] The output, after residual connection and layer normalization, is used as the input for the next layer. The final output is the temporal fusion feature:

[0168]

[0169] The Temporal Transformer module is used to capture the dynamic dependencies between video frames. Through a multi-head attention mechanism, the model can simultaneously focus on local actions and global dynamics, thereby improving the consistency of temporal semantics.

[0170] Step 3: Text Feature Mapping and Cross-Modal Alignment

[0171] like Figure 4 As shown, text features Projected through a fully connected layer to the same embedding space as the video features:

[0172]

[0173] in, This is the text projection matrix. Then, the semantic matching score between the video and the text is calculated using dot product similarity:

[0174]

[0175] This step projects video temporal features and text features into a unified semantic space, and achieves cross-modal semantic alignment through cosine similarity, providing a foundation for subsequent classification and matching tasks.

[0176] Step 4: Semantic Aggregation and Global Feature Generation

[0177] To obtain the overall semantic representation of the video, global average pooling is performed on the temporal feature 𝐹:

[0178]

[0179] Output video-level features This represents the semantic expression of the entire video. Together with the text feature 𝑇′, it forms a cross-modal feature pair, used to calculate the final classification logits or matching score.

[0180] Global pooling operations enable semantic compression of the time dimension, integrating frame-level information into a global semantic representation, which maintains semantic integrity and reduces subsequent computational complexity.

[0181] Step 5: Output after multimodal fusion

[0182] The final output includes three types of results:

[0183] The fused video feature sequence, 𝐹, is used for subsequent classification or action recognition tasks.

[0184] Cross-modal similarity matrix: Used for text and video retrieval;

[0185] Global features of the video: , used as input for downstream models.

[0186] These outputs can be used flexibly in different tasks: if used for action recognition, then... Input classification layer; if used for video retrieval, then utilize... Calculate similarity and sort the results.

[0187] Step 6: Gradient Propagation and Parameter Update

[0188] All modules (AOSM, Transformer, and text mapping layer) participate in end-to-end training, and gradients are backpropagated from the loss function to the parameters of each layer. The gradient calculation path is as follows:

[0189]

[0190] This step ensures that semantic alignment capabilities are progressively enhanced during training, enabling the model to adaptively learn the association patterns between video frames and text descriptions.

[0191] This implementation introduces a temporal Transformer structure to dynamically model inter-frame dependencies and aligns the enhanced features output by the AOSM module with the text semantics in a unified embedding space. Through multi-head attention and global pooling mechanisms, a balance is achieved between temporal continuity and semantic matching accuracy. Figure 3 This demonstrates the structural hierarchy of temporal attention. Figure 4 The calculation process for video-text alignment is demonstrated.

[0192] Specific Implementation Method 4: Model Training Process and Optimization Mechanism

[0193] like Figure 4 As shown, this implementation describes a model training and optimization mechanism based on AOSM-BIKE. The process includes key stages such as training data preparation, model initialization, forward propagation, loss calculation, gradient update, and parameter optimization.

[0194] Step 1: Training Data Preparation

[0195] The input training set source is consistent with Example 1, including video frame sequences, text descriptions, and corresponding label files. Batch loading is used to reduce memory overhead. The input data structure is as follows:

[0196]

[0197] in, Represents a video frame sequence. This indicates the corresponding text. This represents the category label. The data is standardized and randomly augmented before being fed into the model to improve generalization ability.

[0198] This step ensures that each batch contains multimodal samples, meeting the model's requirement for synchronized image and text input. Random augmentation operations include random frame cropping, horizontal flipping, and brightness perturbation to increase sample diversity.

[0199] Step 2: Model Initialization and Parameter Configuration

[0200] Before training begins, the pre-trained Visual Transformer (ViT) and text encoder parameters are loaded. The AOSM module and temporal Transformer parameters are randomly initialized. The main parameter configurations are as follows:

[0201] Visual encoder: ViT-L / 14

[0202] Text Encoder: Transformer Text Encoder

[0203] Learning rate: 0.00005

[0204] Optimizer: AdamW

[0205] Loss function: Dual InfoNCE

[0206] The pseudocode for the model initialization phase is shown in Table 1:

[0207] This step reduces training time by loading pre-trained weights while maintaining the learnability of the AOSM module and the Transformer.

[0208] Step 3: Forward Propagation Process

[0209] The complete forward propagation process includes feature extraction, semantic-guided enhancement, and cross-modal fusion:

[0210] Input a video frame sequence and a text description;

[0211] Embedded features were extracted using a visual encoder and a text encoder:

[0212]

[0213] Inputting 𝑉 and 𝑇 into the AOSM module at the frame level weighting or selection:

[0214]

[0215] The enhanced features are then input into the temporal Transformer module:

[0216]

[0217] Finally, global pooling is used to generate global video features:

[0218]

[0219] The forward propagation process spans the visual, semantic, and temporal dimensions, enabling dynamic interaction between modalities. Enhanced features V′ strengthen semantic relevance through the AOSM module, providing the Transformer with more focused feature inputs.

[0220] Step 4: Backpropagation and Gradient Update

[0221] After calculating the loss, the model performs backpropagation to update the parameters. The gradient propagation path is as follows:

[0222]

[0223]

[0224] The pseudocode for gradient update is shown in Table 2:

[0225] The gradient is passed sequentially from the output layer to the Transformer, the AOSM module, and the visual and text encoders. This mechanism ensures that the parameters for semantic weight allocation and feature augmentation are continuously optimized during training.

[0226] Step 5: Model Saving and Deployment

[0227] After the model has finished training, save the following parameters: visual encoder weights; text encoder weights; AOSM module parameters (including temperature parameter τ); and temporal Transformer weights.

[0228] During the deployment phase, the model only requires a sequence of video frames as input to automatically calculate frame weights, extract semantic features, and output classification prediction results or semantic similarity matrices.

[0229] This step enables a seamless transition from model training to inference. During deployment, the AOSM module can be invoked independently or embedded into a complete multimodal inference system.

[0230] This implementation describes the training and optimization process of the AOSM-BIKE model, including data preprocessing, module initialization, forward propagation, loss calculation, gradient update and deployment. The entire process follows a standard deep learning training pipeline, and the parameter settings are clear and fully reproducible. Figure 4 It is evident that the training and optimization modules are independent of each other but logically continuous, laying the foundation for efficient model training and cross-modal semantic consistency.

[0231] Example: To verify the beneficial effects of the present invention, the following experiments were conducted:

[0232] To verify the practical effect of the proposed "text-guided adaptive frame selection and enhancement method," experiments were conducted on the HMDB-51 standard action recognition dataset, focusing on the impact of the frame selection module on semantic alignment, computational efficiency, and model robustness. The experiments employed single-card training with a batch size of 32, a total of 20 epochs, cosine learning rate decay, and cross-entropy loss. CLIP visual backbone parameters were frozen, and only the AOSM, temporal Transformer, and classification layers were trained. After training, the model achieved a Top-1 accuracy of 66.4% on the validation set, demonstrating that the proposed lightweight frame selection module can provide more discriminative video representations for subsequent cross-modal fusion without significantly increasing the number of parameters. This verifies its effectiveness and deployment friendliness in multimodal video understanding tasks.

[0233] The experimental environment used in this embodiment is shown in Table 3:

[0234]

[0235] This invention uses PyTorch as the deep learning framework to implement an action recognition model. Experiments were conducted on a computer equipped with an NVIDIA A40 graphics accelerator (hardware information is based on actual environment records). Training used the AdamW optimizer with an initial learning rate of 0.00005 (5e-5), weight decay of 0.2, batch size of 32, and a total of 20 epochs. The learning rate was scheduled using cosines with 5 warm-up steps. The model used ViT-L / 14 as the backbone, loaded with Kinetics-400 pre-trained weights, and the input was RGB modality with a resolution of 224×224 and 8-segment sampling. The text branch parameters were fixed (fix_text: True), while the video branch participated in training (fix_video: False). The dataset was HMDB51 (51 classes).

Claims

1. Claim 1: A multimodal video understanding method based on adaptive optimization, characterized in that, Includes the following steps: The multimodal feature extraction step is used to extract video frame-level visual features and text semantic features respectively; An adaptive frame selection step is used to calculate the similarity between the frame-level visual features and the text semantic features, and to generate frame attention weights based on learnable temperature parameters. The feature enhancement step is used to perform residual enhancement or Top-K filtering on frame-level visual features based on the frame attention weights to obtain enhanced video features. The classification step is used to feed the enhanced video features into the temporal fusion head to obtain the action classification result.

2. Claim 2: The method according to claim 1, wherein the adaptive frame selection step comprises: L2 normalization was performed on both frame-level visual features and textual semantic features. Calculate the similarity matrix by scaling the dot product attention; Frame attention weights are generated using the Softmax function and a learnable temperature parameter τ.

3. Claim 3: The method according to claim 1 or 2, wherein the feature enhancement step comprises: In weighted mode, frame-level visual features are multiplied element-wise by (1 + attention weight) along the time dimension to perform residual enhancement; In topk mode, the Top-K frames are selected in descending order of attention weight and then reordered according to their original chronological order.

4. Claim 4: The method according to claim 1, wherein the temperature parameter τ is a learnable scalar with an initial value greater than 0, and is updated together with the other parameters of the model during training.

5. Claim 5: The method according to claim 1, wherein the temporal fusion head is a bidirectional interactive knowledge embedding of the original Temporal Transformer or VCS module of the BIKE framework, and the enhanced video feature shape is maintained as [B,T,D] or [B,K,D].

6. Claim 6: The method according to claim 1, further comprising: After the feature enhancement step and before the classification step, a multimodal feature enhancement module (FEM) is introduced to perform gated residual fusion on the enhanced video features using textual semantic features.

7. Claim 7: The method according to claim 1, wherein the classification step employs symmetric cross-entropy loss to simultaneously optimize visual-text bidirectional logits.

8. Claim 8: The method according to claim 1, wherein all steps are performed within a unified end-to-end framework, and gradients are simultaneously fed back to the visual encoder, text encoder, adaptive frame selection module, and time-series fusion head.

9. Claim 9: A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the adaptive optimization-based multimodal video understanding method as described in any one of claims 1 to 8.

10. Claim 10: A video action recognition system, comprising a processor and a memory, the memory storing instructions for executing the adaptive optimization-based multimodal video understanding method as described in any one of claims 1 to 8.

Images