Video behavior identification method based on multi-mode high-order statistical adapter
By using a multimodal high-order statistical adapter network, combined with high-order statistical modeling and spatiotemporal enhancement modules for visual and textual branches, the high computational cost and insufficient recognition accuracy of existing video behavior recognition methods are solved, achieving efficient video behavior recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-27
AI Technical Summary
Existing video behavior recognition methods suffer from high computational costs and catastrophic forgetting problems during efficient parameter fine-tuning, and lack the ability to model high-order statistical features and cross-modal semantic fusion, resulting in insufficient recognition accuracy.
A multimodal higher-order statistical adapter network is adopted. By freezing the basic network parameters, a higher-order statistical adapter for the visual branch and a parallel text adapter for the text branch are introduced. Combined with a spatiotemporal enhancement module and a higher-order statistical modeling module, the inter-frame dynamics and global temporal sequence of video images are captured, and cross-modal feature correlation is enhanced.
It significantly improves the accuracy and generalization performance of video behavior recognition, especially in complex behavior scenarios, where the recognition accuracy reaches 83.2%, while reducing computational costs.
Smart Images
Figure CN121746987A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of video behavior recognition, and in particular to a video behavior recognition method based on a multi-modal high-order statistical adapter. BACKGROUND
[0002] In recent years, video behavior recognition based on contrastive language-image pre-training models has made significant progress, but traditional full-parameter fine-tuning methods face two major challenges: on the one hand, fine-tuning the entire network leads to a dramatic increase in computational cost; on the other hand, changes in original model parameters can easily cause catastrophic forgetting, significantly weakening their generalization ability. Although the rise of parameter-efficient fine-tuning methods improves efficiency by freezing main parameters and only training new components, existing adapter fine-tuning techniques have significant limitations in capturing video context associations.
[0003] Current mainstream long-time dependency modeling methods can be divided into two categories: one is graph convolutional neural networks, which model dependencies through information aggregation of semantically similar nodes, but only act on high-order features, and lack sensitivity to subtle differences; the other is based on self-attention mechanism methods, which can enrich features through pixel-level relationship modeling, but perform poorly in distinguishing subtle differences between similar behaviors. The essential defect of the two methods is that they only focus on first-order information and lack effective modeling of high-order statistical features, making it difficult for the model to accurately capture complex dynamics and subtle differences between classes in videos, limiting its performance improvement in fine-grained behavior recognition tasks.
[0004] In addition, existing multi-modal models often fail to fully utilize the text description information of action labels when fusing text and visual information due to the lack of semantic alignment mechanisms. How to build a video behavior recognition framework with high-order statistical feature modeling capability and cross-modal semantic fusion capability while ensuring parameter efficiency has become a technical problem that needs to be solved. SUMMARY
[0005] According to the problems existing in the prior art, the present application discloses a video behavior recognition method based on a multi-modal high-order statistical adapter, which specifically comprises the following steps: Obtain video data and its corresponding label text description information to establish a text-video dataset, divide the text-video dataset into a training set and a test set, frame the video data and save it; The comparative language image pre-training model is used as a basic network, original double-branch parameters of the basic network are frozen, and the basic network is expanded, wherein the original double-branch of the basic network includes a visual branch and a text branch; a parallel text adapter is inserted into the text branch, and a high-order statistical adapter is inserted into the visual branch, so as to establish a video multi-modal high-order adapter network, which classifies video behaviors by calculating similarity information of video features and text features; the high-order statistical adapter includes a space-time enhancement module and a high-order statistical modeling module; The video multi-modal high-order adapter network is iteratively trained using a training set, so as to update network parameters, wherein the training process includes forward propagation of network features and backward propagation of errors; In each iteration, network parameters are updated and training and verification are performed, optimal weights of the video multi-modal high-order adapter network are saved according to an optimal verification set, the video multi-modal high-order adapter network is initialized using the optimal weights, and performance of the network is evaluated on a test set.
[0006] Further, the space-time enhancement module is used to capture inter-frame dynamics and global timing of video images, and the high-order statistical modeling module is used to mine deep semantic differences of video images; The space-time enhancement module processes features of video images as follows: time dimension strengthening processing of input features of video images is performed through one-dimensional time convolution to obtain global timing correlation, a feature difference value between a current frame and a previous frame is calculated to capture local motion patterns and dynamic differences of adjacent frames, and the dynamic differences are input into three-dimensional space convolution to complete fusion of local dynamics and spatial features, and space-time fusion features are output; The high-order statistical modeling module performs dimension compression on the space-time fusion features output by the space-time enhancement module through three-dimensional convolution downsampling to obtain low-dimensional feature maps.
[0007] Further, the video multi-modal high-order adapter network adopts a double-branch paradigm architecture based on a comparative language image pre-training model, and includes a video encoder and a text encoder, wherein: The video encoder takes a Vision Transformer with frozen pre-training weights as a backbone network, and embeds a high-order statistical adapter, which is used to realize coding and fine-grained modeling of video features; The text encoder takes a Transformer with frozen pre-training weights as a basic architecture, and integrates a parallel text adapter, which is used to complete coding and adaptive optimization of text semantics; The video encoder and the text encoder work cooperatively to respectively perform deep feature learning on video features and text features, and identify video behaviors through semantic correlation of cross-modal features.
[0008] Further, the feature encoding process of the video encoder comprises: Preprocessing the input video, the video containing T video frames, each video frame having a size of HxWx3, each video frame being divided into non-overlapping blocks with a spatial size of PXP, and obtaining a feature sequence after block embedding processing; splicing class labels for learning for the feature sequence, and adding position encoding, to construct the input feature of the t-th frame as:
[0009] Constructing the input feature of the entire video as:
[0010] Inputting the video input feature into a Vision Transformer layer, and outputting a feature as:
[0011] wherein is a high-order statistical adapter.
[0012] Further, the high-order statistical modeling module reconstructs the inter-frame difference feature as , constructs a feature map set , calculates a high-order statistical score based on a standard moment formula after resolving channel correlation through down-sampling convolution, distributes the normalized statistical score to each position of the feature map based on an attention mechanism according to feature affinity, adopts a residual connection to stabilize the training process, outputs a reweighted feature, restores the reweighted feature to the original dimension through three-dimensional convolution up-sampling, and completes high-order statistical feature modeling.
[0013] Extracting the video frame class label in the output of the last layer, projecting it to a video language (VL) space through a projection layer, obtaining , and averaging along the time dimension to generate a final video representation: .
[0014] Further, performing tokenization processing on the input text to obtain a word sequence, generating a feature sequence after word embedding projection, and constructing an input feature of the text encoder as:
[0015] After processing through a Transformer layer, the output text feature is:
[0016] wherein is a parallel text adapter, The self-attention mechanism of the Transformer layer is combined with the text features processed by the feedforward layer. As input, the output features are processed by a parallel text adapter:
[0017] Select the most representative token and project it onto the VL space through a projection layer to obtain the final text representation:
[0018] in, It is the projection layer.
[0019] By employing the aforementioned technical solutions, this invention provides a video behavior recognition method based on a multimodal higher-order statistical adapter. In the visual branch, a higher-order statistical adapter is embedded. The spatiotemporal enhancement module effectively captures long-term dependencies and spatial features of the video, while the higher-order statistical modeling module significantly amplifies subtle differences between similar features by projecting features into a high-dimensional space. Simultaneously, it utilizes partial statistical data for relationship modeling, improving feature compactness and difference capture accuracy. In the text branch, a parallel text adapter is introduced to fully integrate textual and visual information, enhancing the semantic representation capability of action labels. Compared to existing methods, this invention achieves a recognition accuracy of 83.2% on the Kinetics-400 dataset and demonstrates stronger recognition advantages and generalization performance in complex behavioral scenarios. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 The flowchart of the method of the present invention Figure 2 This is a diagram showing the overall structure of the video multimodal high-order adapter network in this invention. Figure 3 This is a diagram of the high-order statistical attention structure in the video multimodal high-order adapter network of this invention. Detailed Implementation To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] like Figure 1 The video action recognition method based on a multimodal high-order statistical adapter, as shown, specifically includes the following steps: S1: Obtain videos and their text descriptions to construct a text-video dataset. The video is frame-stripped and divided into a training set and a test set. The training set is used for model training, and the test set is used for model testing. S11: Obtain the video dataset Kinetics400 (K400) and its textual descriptions of labels, and the composition of the video data. The corresponding text description constitutes The video dataset is divided into a training set and a test set. S12: Divide the text video dataset into a training set and a test set. The training set is used for model training, and the test set is used for model testing. S13: Perform segment sampling on the above video data; S14: Use FFmpeg to extract 8 frames from the sampled video clip. like Figure 2 As shown: S2: Using a contrastive language image pre-trained model as the base network, and freezing the original dual-branch (visual branch, text branch) parameters of the base network, the base network is extended to construct a video multimodal high-order adapter network. The video multimodal high-order adapter network classifies videos based on the similarity information between video features and text features. S21: The video multimodal high-order adapter network consists of two parts: a video encoder and a text encoder.
[0024] S22: For data from text-video datasets A video clip and its corresponding text description Input to the video encoder respectively and text encoder middle, Depend on A Vision Transformer layer and higher-order statistical adapters composition, Depend on Transformer layer and parallel text adapter Composition. Obtaining video and text features:
[0025]
[0026] To obtain the final video representation Extract the last layer Output Class tags of video frames Projected into Video Language (VL) space ,in, It is a projection layer. Averaging along the time dimension yields the final video representation:
[0027] Text representation of the final VL space Depend on It is concluded that, among them As the most representative token, It is the projection layer.
[0028] S23: The model uses cosine similarity to calculate video features. and enhance text features Similarities between them:
[0029] As shown in Figure 2: S3: In the higher-order statistical adapter of the vision branch, two functional modules are designed to address the two core requirements of video temporal modeling and higher-order feature mining. One is a spatiotemporal enhancement module for capturing inter-frame dynamics and global temporal sequence, and the other is a higher-order statistical modeling module for mining deep semantic differences. S31: Preprocess the sampled frames of the input video. First, divide the frame data into non-overlapping feature blocks and map them into block embeddings. Then, adjust the feature dimensions to make the input features fit the module input requirements of the high-order statistical adapter, providing basic features for subsequent spatiotemporal and high-order feature modeling. S32: The feature processing flow of the spatiotemporal enhancement module is as follows: First, the input features are enhanced temporally through one-dimensional temporal convolution to highlight global temporal correlation; then, the feature difference between the current frame and the previous frame is calculated using the temporal difference module (inter-frame subtraction operation) to accurately capture the local motion patterns and dynamic differences between adjacent frames; then, the difference features are input into three-dimensional spatial convolution to complete the fusion of local dynamics and spatial features, and output spatiotemporal fusion features. S33: The feature processing flow of the high-order statistical modeling module is as follows: First, the spatiotemporal fusion features output by the spatiotemporal enhancement module are compressed in dimension through 3D convolutional downsampling to obtain a low-dimensional feature map; then, the low-dimensional feature map is input into the high-order statistical attention module, which includes two core operations: statistical score calculation and statistical score distribution. like Figure 3 As shown: 1) Calculate the statistical score: Reconstruct the inter-frame difference features output by the temporal difference module into... Construct a set of feature maps After dispersing channel correlation through 1×1×1 downsampling convolution, higher-order statistical scores are calculated based on the standard moment formula, and the k-th standard moment is defined as:
[0030] in, It is a set The mean and standard deviation, It is a set The number of elements; select As a subset of statistical scores, further, a functional function Used to calculate sets middle Required statistical score:
[0031] in, yes middle Statistical scores, for example, yes The mean; 2) Distribution of Statistical Scores: Based on the attention mechanism, the standardized statistical scores are distributed to various locations on the feature map according to feature affinity through the statistical score distribution module. Residual connections are introduced to stabilize training and output reweighted features. Finally, 3D convolutional upsampling is used to restore the reweighted features to the original dimension, completing the high-order statistical feature modeling, which can be represented as:
[0032] in, For concatenating output features, yes The concatenation of statistical vectors, where yes The number of elements.
[0033] To integrate the higher-order statistical attention module into existing neural network models, we implemented residual connections to reduce covariance shift and facilitate gradient updates. Ultimately, our higher-order statistical attention module is defined as:
[0034] in, and These represent the feature maps of the input and output, respectively. It is a 1×1×1 convolution. It is a batch normalization layer to restore the features to the number of channels of the first input features; S34: The final output feature of the high-order statistical adapter is the high-order statistical fusion feature output by the high-order statistical modeling module. This feature integrates global temporal correlation, local inter-frame dynamics and high-order semantic difference information. It can be directly input into subsequent visual branch network layers to achieve fine-grained modeling of video visual features.
[0035] As shown in Figure 2: S4: To address the problem that text label descriptions in behavior recognition are short and that the text encoder of the contrastive language image pre-trained model has difficulty effectively distinguishing semantic features of similar actions, a parallel text adapter is introduced in the text branch, and combined with a video-guided text expansion strategy, the semantic representation learning and cross-modal alignment of video labels are optimized. S41: Parallel Text Adapter Architecture Design: Parallelization improvements based on adapter strategy, including self-attention mechanism in the Transformer layer and text features processed by the feedforward layer. As input, the output features are processed by a parallel text adapter:
[0036] in, and These represent the downsampling layer and the upsampling layer, respectively. and It is a non-linear activation function; this structure enhances the ability to extract semantic features from short text tags by parallelizing the feature transformation path; S42: Semantic Enhancement Strategy: The video-guided text expansion method is adopted to integrate video caption information into the original short text labels. By expanding the semantic dimension of the labels, the model's understanding of the behavior labels is enhanced. This strategy enables text features and visual features to achieve more accurate alignment in the semantic space, thereby improving the accuracy and semantic matching of video action recognition. S5: Iteratively train the video multimodal high-order adapter network using text and video data from the training set to update the network parameters. This training process includes forward propagation of network features and backward propagation of errors. During implementation, a Transformer was used as the text encoder in the video multimodal higher-order adapter network, and a ViT-B / 16 was used as the video encoder. The original dual-branch (visual branch, text branch) parameters of the base network were frozen, and the base network was extended. Each sampled frame was cropped to... The shape is shown. The dataset was trained for 15 iterations on the K400 dataset using the AdamW optimizer with an initial learning rate of 3e-4 and a weight decay factor of 0.01.
[0037] S6: Update the network parameters in each iteration, perform training and validation in each iteration, save the optimal weights of the video multimodal high-order adapter network based on the optimal validation set accuracy, initialize the video multimodal high-order adapter network with the optimal weights, and evaluate the performance of the network on the test set.
[0038] During implementation, the following methods were adopted: The strategy was tested. Top-1 recognition accuracy (%) and model computational cost GFLOPs were used as evaluation criteria.
[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A video behavior recognition method based on a multimodal high-order statistical adapter, characterized in that... include: A text video dataset is created by acquiring video data and its corresponding text description information. The text video dataset is divided into a training set and a test set. Frames are extracted from the video data and saved. Using a contrastive language image pre-trained model as the base network, the original bi-branch parameters of the base network are frozen, and the base network is extended, wherein the original bi-branch of the base network includes a visual branch and a text branch. A parallel text adapter is inserted into the text branch, and a higher-order statistical adapter is inserted into the vision branch, thereby establishing a video multimodal higher-order adapter network. This video multimodal higher-order adapter network classifies video behaviors by calculating the similarity information between video features and text features. The higher-order statistical adapter includes a spatiotemporal enhancement module and a higher-order statistical modeling module. The video multimodal high-order adapter network is iteratively trained using the training set to update the network parameters. The training process includes forward propagation of network features and backpropagation of errors. In each iteration, the network parameters are updated and training and validation are performed. The optimal weights of the video multimodal high-order adapter network are saved based on the optimal validation set. The video multimodal high-order adapter network is initialized with the optimal weights, and the performance of the network is evaluated on the test set.
2. The video behavior recognition method based on a multimodal high-order statistical adapter according to claim 1, characterized in that: The spatiotemporal enhancement module is used to capture the inter-frame dynamics and global temporal sequence of video images, and the high-order statistical modeling module is used to mine the deep semantic differences of video images. The spatiotemporal enhancement module processes the features of the video image as follows: it enhances the input features of the video image temporally through one-dimensional temporal convolution to obtain global temporal correlation, then calculates the feature difference between the current frame and the previous frame, captures the local motion patterns and dynamic differences of adjacent frames, inputs the dynamic differences into three-dimensional spatial convolution, completes the fusion of local dynamics and spatial features, and outputs spatiotemporal fusion features. The higher-order statistical modeling module compresses the spatiotemporal fusion features output by the spatiotemporal enhancement module through three-dimensional convolutional downsampling to obtain a low-dimensional feature map.
3. The video behavior recognition method based on a multimodal high-order statistical adapter according to claim 1, characterized in that: The video multimodal high-order adapter network adopts a dual-branch paradigm architecture based on a contrastive language image pre-trained model, including a video encoder and a text encoder, wherein: The video encoder uses a Vision Transformer with frozen pre-trained weights as its backbone network and embeds a high-order statistical adapter to encode video features and perform fine-grained modeling. The text encoder is based on a Transformer architecture with frozen pre-trained weights and integrates a parallel text adapter to perform text semantic encoding and adaptation optimization. The video encoder and text encoder work together to perform deep representation learning on the features of video and text modalities respectively, and identify video behaviors through semantic association of cross-modal features.
4. The video behavior recognition method based on a multimodal high-order statistical adapter according to claim 3, characterized in that, The feature encoding process of the video encoder includes: The input video is preprocessed, consisting of T video frames, each with a size of H×W×3. Each video frame is divided into non-overlapping blocks, where the block size is P×P. After block embedding, a feature sequence is obtained. Class labels for learning are concatenated to the feature sequence, and positional encoding is added to construct the input features for the t-th frame: The input features for constructing the entire video are as follows: The video input features are input into the Vision Transformer layer, and the output features are: in It is a high-order statistical adapter.
5. The video behavior recognition method based on a multimodal high-order statistical adapter according to claim 4, characterized in that, The higher-order statistical modeling module reconstructs the inter-frame difference features as follows: Constructing a set of feature maps After dispersing channel correlations through downsampling convolution, higher-order statistical scores are calculated based on the standard moment formula. Then, based on an attention mechanism, the standardized statistical scores are distributed to various locations in the feature map according to feature affinity. Residual connections are used to stabilize the training process, and reweighted features are output. 3D convolutional upsampling restores the reweighted features to their original dimensions, completing the higher-order statistical feature modeling. Extract the video frame class labels from the last layer output, and project them onto the video language space through a projection layer to obtain... And averaged along the time dimension to generate the final video representation: 。 6. The method according to claim 3, characterized in that, The feature encoding process of the text encoder includes: tokenizing the input text to obtain a word sequence, generating a feature sequence after word embedding projection, and constructing the input features of the text encoder as follows: After processing by the Transformer layer, the output text features are: in, For parallel text adapters, The self-attention mechanism of the Transformer layer is combined with the text features processed by the feedforward layer. As input, the output features are processed by a parallel text adapter: Select the most representative token and project it onto the VL space through a projection layer to obtain the final text representation: in, It is the projection layer.