Efficient micro gesture recognition method based on motion-aware state fusion manhattan model

Abstract

The application discloses a high-efficiency micro gesture recognition method based on motion perception state fusion of a manba model, and relates to the technical field of micro gesture recognition. The method comprises the following steps: data preprocessing, micro gesture video preprocessing, conversion into a space-time token sequence and embedding of position information; feature extraction: for the embedded sequence, a bidirectional state space model is used to capture global space-time dependence, and global space-time features in the form of a hidden state sequence are generated; then, a multi-scale center frame difference state fusion module is used to restore the hidden state sequence to a space-time structure, and motion perception features are captured by calculating the difference between the current frame and the adjacent frame through a center frame difference operation; the restored global space-time features and the motion perception features are subjected to multi-scale local fusion, and finally, a multi-scale feature set is output; multi-scale feature fusion; classification prediction; model training and optimization; and model reasoning. Through adaptive fusion of multi-scale information, the application can accurately capture subtle changes in micro gestures.

Description

Technical Field

[0001] This invention belongs to the field of micro-pose recognition technology, and particularly relates to an efficient micro-pose recognition method based on motion-sensing state fusion Mamba model. Background Technology

[0002] Micro-gesture recognition is a key task in computer vision and video analytics, aiming to identify subtle, delicate, and often unconscious human movements, such as "crossing fingers" or "touching the chin." This technology plays a crucial role in applications such as human-computer interaction and emotion understanding. However, the characteristics of micro-gestures—small amplitude, short duration, and subtle morphological changes—pose significant challenges to recognition models. These models not only need to capture fine local spatiotemporal dependencies to perceive minute movements but must also possess the ability to model long-range global contexts to understand the complete semantics of the action.

[0003] In the history of technological development, models based on Convolutional Neural Networks (CNNs), such as C3D and I3D, have been widely adopted. These models effectively extract spatiotemporal features from videos through the operation of their convolutional kernels within local receptive fields. However, their inherent locality also constitutes a major limitation, namely, the difficulty in modeling global dependencies between distant frames or pixels in a video sequence, resulting in poor performance for micro-pose recognition that requires long-range contextual understanding. To overcome the global modeling shortcomings of CNNs, Transformer-based models, such as TimeSformer and VideoSwinTransformer, have been introduced into this field. These models utilize self-attention mechanisms to effectively capture global dependencies, but their computational complexity increases quadratically with the length of the input sequence, leading to high computational costs and memory overhead, severely restricting their practical deployment in real-time or lightweight micro-pose recognition scenarios.

[0004] In recent years, state-space models (SSMs), especially the Mamba model, have attracted attention for their ability to combine linear computational complexity with strong long-range dependency capture in long sequence modeling. Subsequent video Mamba models have attempted to apply this architecture to video understanding. However, directly applying the original Mamba model to micro-pose recognition has significant shortcomings: First, the Mamba is essentially a one-dimensional sequence model, which flattens the spatiotemporal blocks of video into a one-dimensional sequence and processes them in scanning order. State updates depend only on the previous state. This mechanism destroys the inherent three-dimensional local structure of video data, resulting in a lack of explicit modeling ability for local spatiotemporal dependencies. Although some works have attempted to introduce locality through different scanning strategies, this problem has not been addressed within the core state space. Second, existing methods generally lack a perception mechanism specifically for "motion," the core element of micro-pose recognition. The model struggles to explicitly and effectively capture subtle dynamic changes between frames, thus limiting its ability to distinguish micro-poses that appear similar but have different motion patterns.

[0005] Therefore, existing technologies face a dilemma: convolutional neural network models have strong local capabilities but weak global capabilities; Transformer models have strong global capabilities but low efficiency; and while the emerging Mamba model is efficient and can handle long-range dependencies, it has significant shortcomings in local spatiotemporal modeling and explicit motion perception. Currently, there is an urgent need for a micro-pose recognition scheme that can simultaneously achieve efficient computation, powerful long-range dependency modeling, refined local spatiotemporal perception, and utilization of explicit motion cues.

[0006] Based on the in-depth analysis of existing models in the background technology, this invention aims to solve the following key technical problems exposed when applying state-space models to micro-pose recognition tasks: (1) Existing video Mamba models treat videos as one-dimensional sequences, and their core state update mechanism only depends on the previous state in the sequence, which cannot effectively perceive and model the inherent local spatiotemporal structure in video data. This makes the model insensitive to capturing patterns in micro-poses that depend on subtle changes in specific body parts within a short time series. (2) Existing methods generally rely on implicitly learning motion features from the original frames, lacking an explicit module specifically for emphasizing and capturing subtle dynamic changes between frames. This makes it difficult for the model to focus on the core of micro-poses—motion information, thus limiting its ability to distinguish micro-poses that are similar in appearance but have different motion patterns. (3) Different micro-poses are diverse in spatiotemporal range, and existing models lack a mechanism that can adaptively fuse spatiotemporal context information at different scales, and cannot dynamically adjust the perception range according to the specific pose, resulting in limited representation capabilities for certain types of micro-poses.

[0007] Therefore, the fundamental objective of this invention is to provide a new technical solution, specifically an efficient micro-pose recognition method based on the motion-aware state fusion Mamba model. While inheriting the advantages of the state-space model in efficiently processing long sequences, this method effectively enhances its local spatiotemporal modeling capability and motion-awareness capability, and achieves adaptive fusion of multi-scale information. Ultimately, this provides a micro-pose recognition method that is superior to existing technologies in both recognition accuracy and computational efficiency. Summary of the Invention

[0009] The purpose of this invention is to provide an efficient micro-pose recognition method based on a motion-aware state fusion Mamba model, in order to solve the problems that existing technical solutions (convolutional neural networks, Transformer, basic Mamba) proposed in the background art cannot achieve high performance in micro-pose recognition tasks while maintaining high efficiency, and simultaneously achieving powerful long-range dependency modeling, fine local spatiotemporal perception, and explicit motion cue utilization.

[0010] To achieve the above objectives, the present invention employs the following technical solution: In its first aspect, this invention proposes an efficient micro-pose recognition method based on a motion-aware state fusion Mamba model, comprising the following steps: S1. Data preprocessing: Frame sampling, size normalization, and data augmentation are performed on the micro-pose video. Then, the video frames are converted into spatiotemporal token sequences and embedded with position information. S2. Feature Extraction: For the embedded sequence, a bidirectional state space model is used to capture the global spatiotemporal dependency and generate global spatiotemporal features of the hidden state sequence. Then, the multi-scale center frame differential state fusion module is used to restore the spatiotemporal structure of the hidden state sequence. The difference between the current frame and the adjacent frames is calculated through the center frame differential operation to explicitly capture motion-aware features. The restored global spatiotemporal features and motion-aware features are fused at multiple scales locally, and finally, a multi-scale feature set is output. S3. Multi-scale feature fusion: An adaptive scale weighting module is used to dynamically aggregate multi-scale features to obtain the final fused features. S4. Classification Prediction: The classification prediction module performs global aggregation and linear classification on the final fused features, and outputs the micro-pose category prediction results. S5. The micro-pose recognition model, consisting of a bidirectional state space model, a multi-scale center frame differential state fusion module, an adaptive scale weighting module, and a classification prediction module, is trained and optimized end-to-end using the cross-entropy loss function. S6. Use the trained and optimized micro-pose recognition model for model inference.

[0011] Preferably, in step S1, the video frames are converted into spatiotemporal token sequences by performing 3D block embedding processing on the video frames; the 3D block embedding processing is specifically implemented by using a 3D convolutional layer.

[0012] For a size of The input video clip is converted into a spatiotemporal token sequence using 3D convolutional layers. ,in , in The embedding dimension can be set to 128, 256, or 512 depending on the model size.

[0013] Furthermore, in step S1, video frames are converted into spatiotemporal token sequences and location information is embedded, as follows: First, the preprocessed video frames are segmented without overlap using 3D convolutional kernels, converting them into a spatiotemporal patch sequence. Then, each spatiotemporal patch is mapped to a feature token using a linear layer, resulting in a spatiotemporal token sequence. This is followed by a sequence of spacetime tokens. Add learnable spatial embeddings and time location embedding Spatial location embedding Used to encode the location information of each spatial block; temporal location embedding Used to encode location information for each time step. Both are extended to include spatiotemporal token sequences via a broadcast mechanism. For identical shapes, perform element-wise addition: The input sequence forms the model; the final embedding sequence is This sequence preserves both spatiotemporal features and location information.

[0014] Preferably, the bidirectional state-space model in S2 is as follows: The bidirectional state-space model comprises two parallel branches: a forward state-space model and a backward state-space model. Both the forward and backward state-space models employ a discrete state-space model structure. The forward state-space model considers the input sequence... Sequential processing is performed; the inverse state-space model, on the other hand, processes the sequence after flipping it in the time dimension. Process it.

[0015] Both the forward state-space model and the reverse state-space model contain independent parameters, with the core parameter being the state transition matrix. Input matrix Output matrix ,in N To hide the state dimension, a zero-order preservation method is used to discretize the continuous state-space model, resulting in the discrete matrix: ,in Let I be the discretization step size, and let I be the identity matrix.

[0016] Furthermore, the processing procedure for the bidirectional state-space model is as follows: Embedded sequence Input to the bidirectional state-space model module to generate hidden states with global context enhancement; First, bidirectional feature calculation: The forward state-space model inputs the embedded sequence Z in its original time order and updates it according to the discrete state update formula. , For the first Time tokens generate positive hidden states The inverse state-space model is obtained by performing time flipping on the embedded sequence Z. Generate reverse hidden state And then Flip to get ; Subsequent bidirectional feature fusion: fusion through averaging operations To obtain global spatiotemporal features .

[0017] Preferably, the multi-scale center frame differential state fusion module in S2 includes state reshaping operation, center frame differential operation, multi-scale state fusion mechanism and feature synthesis mechanism; The state reshaping operation reshapes the one-dimensional hidden state sequence into a structured four-dimensional tensor as the original feature, which facilitates subsequent convolution operations targeting local spatiotemporal patterns.

[0018] The center frame difference operation explicitly calculates the dynamic changes between adjacent frames to generate motion-aware features. This operation is one of the core innovations of this invention; it explicitly captures motion information by calculating the differences between the features of the current frame and adjacent frames. Compared to simple first-order difference, center frame difference can more effectively suppress static background interference and highlight the subtle dynamic changes unique to micro-poses. This explicit motion-aware mechanism significantly enhances the model's sensitivity to minute movements.

[0019] The described multi-scale state fusion mechanism applies a multi-scale state fusion operator to locally weight and aggregate the original features and motion-aware features within 3D neighborhoods of different sizes. By defining 3D neighborhoods of different scales, multi-scale state fusion enables the model to simultaneously capture local details and broader contextual information. This multi-scale design allows the model to adapt to micro-pose patterns across different spatiotemporal ranges, effectively solving the problem that a single receptive field cannot adequately handle various action scales.

[0020] The described feature synthesis mechanism, for each scale, linearly combines the fusion results of the original features and the fusion results of the motion features using learnable gating parameters to generate multi-scale motion-aware fusion features. The learnable gating parameters enable adaptive fusion of the original features and motion-aware features. This design allows the model to dynamically adjust the contribution of motion information based on the specific input content, achieving intelligent weighted combination of appearance features and motion cues, significantly improving the flexibility of feature representation.

[0021] Furthermore, the processing procedure of the multi-scale center frame differential state fusion module is as follows: First, feature dimension reshaping: the global spatiotemporal features output by the bidirectional state-space model. Remodeling into 4D tensors Recovery time T and space Dimensional structure; to facilitate subsequent convolution operations targeting local spatiotemporal patterns; Next, the center frame difference is calculated: the center frame difference is calculated in the time dimension. :

[0022] in, Finally, motion perception features are obtained. ; Subsequently, multi-scale local fusion is implemented: three parallel scale branches are set up, and each branch processes F and D through a local weighted aggregation operation.

[0023] in, For 3D neighborhood, Corresponding window radii of 1, 2, and 3, Learnable kernel weights; Finally, feature fusion and output: Introducing learnable gating parameters. Integrating original features with motion-sensing features: This yields a multi-scale feature set.

[0024] Preferably, the adaptive scale weighting module in S3 includes multi-scale feature splicing and stacking operations, a lightweight 3D convolutional attention sub-network, and a weighted summation operation; The multi-scale feature splicing and stacking operations input multi-scale motion perception fusion features into the adaptive scale weighting module, forming a unified multi-scale representation through splicing and stacking operations. By organizing multi-scale features through both splicing and stacking, a structured input is constructed for subsequent adaptive fusion. This dual recombination strategy ensures the complete preservation and effective integration of multi-scale information.

[0025] The lightweight 3D convolutional attention subnetwork calculates attention weights at different scales for each spatiotemporal location. The attention weight calculation employs a lightweight 3D convolutional network to achieve adaptive selection between scales. This module can dynamically allocate attention weights at different scales based on the local characteristics of each spatiotemporal location. This fine-grained weighting mechanism allows the model to autonomously select the most relevant receptive field range.

[0026] The weighted summation operation utilizes calculated attention weights to weight and sum the stacked multi-scale features, generating the final feature map. This weighted fusion process performs the final multi-scale feature integration, aggregating the most representative feature information into a unified representation through the guidance of attention weights. This intelligent fusion method significantly enhances the discriminative power of the final features.

[0027] Furthermore, the processing procedure of the adaptive scale weighting module is as follows: Feature concatenation: Concatenates multi-scale features along the channel dimension to obtain a unified feature. ,Will Stacked as ; Attention weight calculation: based on unified features The scale attention weights are generated using a two-layer 3D convolutional attention subnetwork, and then obtained by softmax normalization. ; Weighted fusion: Through broadcasting mechanism and attention weight Sum of the following dimensions after multiplication:

[0028] Obtain the final fusion features .

[0029] Preferably, the classification prediction module in S4 is specifically as follows: The classification prediction module includes a global aggregation module and a linear classifier; The final fused features are flattened and subjected to global average pooling to aggregate global information and obtain a global feature vector. The global feature vector is then input into a linear classifier to obtain the prediction result of the micro-pose category.

[0030] In the global aggregation module, the final fused features are... Flatten the vector along the spatiotemporal dimension into a one-dimensional vector, and then aggregate all spatiotemporal location information through global average pooling to obtain a global feature vector. ; In the linear classifier, The input linear layer is mapped to the predicted logarithm in the dimension of the number of micro-pose categories.

[0031] Classification preprocessing achieves efficient aggregation of spatiotemporal information through global average pooling. This operation is not only computationally efficient but also robust to changes in the length of the input sequence.

[0032] Preferably, the model inference in S6 adopts the following decision rule:

[0033] in, This represents the total number of micro-pose categories. It is a predicted category. The probability of selecting the category with the highest predicted probability as the final micro-pose recognition result for the video segment ensures that the entire system can learn collaboratively with a unified goal.

[0034] In a second aspect, this invention proposes an efficient micro-pose recognition system based on a motion-aware state fusion Mamba model, comprising: The data preprocessing module performs frame sampling on the micro-attitude video to be tested, and converts the video frames into a spatiotemporal token sequence and embeds position information. The micro-pose recognition model includes a bidirectional state space model, a multi-scale center frame differential state fusion module, an adaptive scale weighting module, and a classification prediction module. The output selects the category with the highest prediction probability as the final micro-pose recognition result of the micro-pose video to be tested.

[0035] Compared with the prior art, the beneficial effects of the present invention are: (1) The method in this invention aims to make up for the shortcomings of the prior art, that is, by introducing local state fusion and explicit motion perception mechanism, the micro-pose recognition method based on state space model is enhanced in terms of local spatiotemporal modeling and dynamic information capture, and the robustness of the model to multiple sample poses is improved by multi-scale fusion and adaptive weighting strategy.

[0036] (2) The method in this invention differs from the previous method of directly using the basic state space model for video understanding. It explicitly enhances the local spatiotemporal modeling capability of the model through the motion perception state fusion mechanism, and injects dynamic motion information through the center frame difference operation, so that the model can accurately capture subtle motion changes in micro-pose, thereby greatly improving the recognition accuracy while maintaining linear computational complexity.

[0037] (3) The method in this invention differs from previous recognition methods that rely on large-scale pre-training or complex attention mechanisms. It innovatively designs a multi-scale local fusion and adaptive weighted architecture, which achieves collaborative modeling of global and local features with minimal computational overhead. This design enables the model to achieve excellent performance on micro-pose recognition tasks without relying on massive pre-training data or high computational costs, providing a practical technical solution for resource-constrained scenarios.

[0038] (4) The method in this invention differs from previous strategies that extract features at a single scale or use fixed weight fusion. It innovatively achieves multi-scale spatiotemporal perception and dynamic adaptive fusion. By processing the spatiotemporal context of different receptive fields in parallel and dynamically integrating multi-scale features using an attention mechanism, the model can autonomously select the most suitable feature granularity according to the characteristics of specific micro-poses, which significantly enhances the generalization recognition ability of micro-poses at different scales and speeds.

[0039] In summary, by organically combining the above-mentioned technological innovations, this invention successfully solves key technical challenges in micro-pose recognition, such as the difficulty in capturing subtle movements, the difficulty in modeling local dependencies, and the difficulty in adapting to multi-scale patterns. It achieves an optimal balance between accuracy, efficiency, and practicality, providing a reliable technical foundation for the practical application of micro-pose recognition technology. Attached Figure Description

[0040] Figure 1 This is a structural block diagram of the micro-pose recognition model in this invention; Figure 2 This is a schematic diagram illustrating the qualitative visualization results of the activation maps output by the method of the present invention under different micro-pose categories; Figure 3 This is a schematic diagram showing the visualization results of the attention weights learned by the method of the present invention at different fusion scales under different micro-pose categories. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1: This invention presents an efficient micro-pose recognition method based on a motion-aware state fusion Mamba model. Unlike previous convolutional neural networks, Transformers, or state-space models, this invention aims to overcome existing limitations by enhancing the modeling of local spatiotemporal dependencies and introducing explicit motion cues. Specifically, see [link to relevant documentation]. Figure 1This invention first captures the global context of a video sequence using a bidirectional state-space model; then, after reshaping the hidden states, it explicitly models the local spatiotemporal pattern and injects motion information using a multi-scale center frame differential state fusion module; finally, it dynamically fuses multi-scale features using an adaptive scale weighting module. The final enhanced feature representation is used for micro-pose classification during the recognition stage. The specific method includes the following steps: Step 1: Data Preprocessing. The input micro-pose video undergoes frame sampling, size normalization, and data augmentation. Simultaneously, spatiotemporal patches are extracted and embedded with location information, laying the foundation for subsequent feature extraction.

[0043] 1) Micro-posture video data preprocessing: We selected publicly available micro-pose datasets (iMiGUE and SMG) as experimental data. iMiGUE contains 18,499 samples (32 micro-pose classes, resolution 1280×720, average duration 2.6 seconds), and SMG contains 3,712 samples (17 micro-pose classes, resolution 1920×1080, average duration 1.8 seconds). The specific preprocessing operations are as follows: First, frame sampling was performed. Each video was evenly divided into 8 groups along the time dimension, and 2 frames were randomly selected from each group, resulting in a total of 16 frames to form a video clip, denoted as the input tensor. Then, the video frames are standardized in size. During the training phase, random cropping is used to adjust the frame size to 224×224, and during the testing phase, center cropping is used to adjust it to 288×288, eliminating the impact of size differences on the model. Micro-gestures are short in duration (usually 1-3 seconds), and uniformly grouping and sampling 16 frames can completely cover the start, process, and end stages of the action, avoiding the loss of key action information due to incomplete frame sampling.

[0044] 2) Block embedding and spatiotemporal location coding: The preprocessed video clips are processed using a 3D convolution kernel with time × height × spatial width. Non-overlapping segmentation is performed with a stride consistent with the kernel size, converting the video into a spatiotemporal patch sequence. Then, linear projection is applied, mapping each spatiotemporal patch to a dimension through a linear layer. d The characteristic token is used to obtain the token sequence. ,in Next, location embedding is added, introducing time-location encoding. Spatial location coding The time location code is as follows: Spatial location coding A learnable parameter matrix is ​​used to adaptively capture spatial location associations through training. The final embedded sequence is... This sequence preserves both spatiotemporal features and location information.

[0045] 3D patch segmentation can integrate video "temporal-spatial" information to capture the spatiotemporal correlation of micro-poses and avoid information fragmentation. Position embedding can solve the "disorderliness" problem of deep learning models. By preserving the temporal logic and spatial location features of micro-poses, it helps the model accurately distinguish fine-grained action differences, thereby improving the class discrimination ability.

[0046] Step 2: Feature Extraction. A bidirectional state-space model is used to capture global spatiotemporal dependencies, and then a multi-scale center frame differential state fusion module is used to inject local motion-aware features.

[0047] 1) Extracting global spatiotemporal features using a bidirectional state-space model: Specifically, the bidirectional state-space model structure design involves constructing two parallel branches: a forward state-space model and a reverse state-space model, both implemented based on the discrete state-space model. The core parameters include the state transition matrix. Input matrix Output matrix ( N To hide the state dimension, and d Consistent).

[0048] Discretization: The zero-order preserve (ZOH) method is used to discretize the continuous state-space model, resulting in the discrete matrix: ,in I is the discretization step size (a learnable parameter), and I is the identity matrix.

[0049] The overall processing procedure for the bidirectional state-space model is as follows: Bidirectional feature computation: The forward state-space model inputs the embedded sequence Z in its original time order and updates it according to the discrete state formula. ( For the first Time tokens generate positive hidden states The inverse state-space model then performs a time flip on Z to obtain... Input the reverse SSM module to generate the reverse hidden state. And then Flip to get Ensure alignment with the forward sequence.

[0050] Bidirectional feature fusion: Fusion of positive and negative features through averaging. Obtain global spatiotemporal features It can simultaneously capture the spatiotemporal dependencies of the past and the future.

[0051] The bidirectional state-space model, through the fusion of forward and backward features, can fully utilize the spatiotemporal context before and after micro-gesture actions to avoid misjudgments caused by the lack of unidirectional information. Its discretization process allows the model to maintain... It achieves linear computational complexity and improves inference speed compared to Transformer when processing 16 frames of micro-pose video to meet the needs of real-time scenarios. Furthermore, the generated global spatiotemporal features can capture long-range dependencies of micro-pose and avoid the problem that traditional convolutional neural networks cannot capture global action correlations due to their limited receptive field.

[0052] 2) The multi-scale center frame differential state fusion module extracts multi-scale motion sensing features: Feature Dimension Reshaping: Reshaping the global spatiotemporal feature H into a 4D tensor Recovery time T and space Dimensional structure.

[0053] Center frame difference calculation: The center frame difference is calculated in the time dimension to highlight the micro-attitude dynamic features. The formula is as follows: ,in Boundary frames First-order difference completion Finally, the motion perception tensor is obtained. The center frame difference operation is one of the core innovations of this invention. This operation explicitly captures motion information by calculating the difference between the features of the current frame and adjacent frames. Compared with simple first-order difference, center frame difference can more effectively suppress static background interference and highlight the subtle dynamic changes unique to micro-poses. This explicit motion perception mechanism significantly enhances the model's sensitivity to minute movements.

[0054] Multi-scale local fusion: Three parallel scale branches are set up (window sizes 3×3×3, 5×5×5, and 7×7×7), and each branch processes F and D through a local weighted aggregation operation.

[0055] in, For 3D neighborhood, Corresponding window radii of 1, 2, and 3, These are learnable kernel weights.

[0056] Feature fusion and output: Introducing learnable gating parameters It integrates original features with motion features: To obtain a multi-scale feature set .

[0057] In the multi-scale center frame difference state fusion module, center frame difference, by calculating the difference between the center frame and adjacent frames, can highlight subtle dynamic changes in micro-pose and suppress static background interference, allowing the model to focus on the action itself. The multi-scale window design can adapt to micro-poses of different amplitudes (small windows capture fine movements, large windows capture coordinated limb movements) to avoid the problem of insufficient coverage by a single scale, and can learn gating. It can also adaptively adjust the contribution weights of motion features to improve the model's adaptability to different types of micro-poses.

[0058] Step 3: Multi-scale fusion. Dynamic aggregation of multi-scale features is achieved through an adaptive scale weighting module.

[0059] In this embodiment, the present invention specifically employs an adaptive scale weighting module to dynamically aggregate multi-scale features. The overall process is as follows: Feature concatenation: Concatenates multi-scale features along the channel dimension to obtain a unified feature. .

[0060] Attention weight calculation: Scale attention weights are generated using a two-layer 3D convolutional subnetwork. First convolutional layer: 3 input channels. d The first layer has 64 input channels and a kernel size of 3×3×3, using the ReLU activation function. The second layer has 64 input channels and 3 output channels (corresponding to 3 scales), with a kernel size of 1×1×1. The output weights are obtained by Softmax normalization. The formula is:

[0061] Where A is the output of the second convolutional layer, and k is the scale index.

[0062] Weighted fusion: stacking multi-scale features into Through broadcasting mechanism and Sum of the following dimensions after multiplication:

[0063] Obtain the final fusion features .

[0064] Compared to the average weighting of traditional multi-scale fusion, the adaptive scale weighting module can ensure that the model focuses on the most relevant scale features of the current micro-pose by dynamically allocating attention weights. It can also use 3D convolutional sub-networks to calculate weights with spatiotemporal information to avoid insufficient temporal adaptability. Furthermore, it can reduce the interference of redundant scale information and improve feature purity and subsequent classification accuracy through dynamic weighting.

[0065] Step 4: Classification and Prediction. The classification and prediction module performs global aggregation and linear classification on the fused features, and outputs the micro-pose category prediction results.

[0066] In this embodiment, the overall processing procedure of the classification prediction module is as follows: Feature flattening and pooling: Flatten the vector along the spatiotemporal dimension into a one-dimensional vector, and then aggregate all spatiotemporal location information through global average pooling to obtain a global feature vector. .

[0067] Linear classifier design: The input linear layer is mapped to the predicted logarithm of the micro-pose category number dimension (32 classes in the iMiGUE dataset, 17 classes in the SMG dataset), as shown in the formula. ,in (C represents the number of categories) These are the classifier parameters.

[0068] This classification prediction module utilizes global average pooling to aggregate features from the entire spatiotemporal sequence of micro-poses to avoid misclassification caused by missing local features. The linear classifier with a small parameter size can reduce the risk of overfitting and lower computational costs.

[0069] Step 5: Model Training and Optimization. Optimize the cross-entropy loss function to improve the model's recognition accuracy.

[0070] This invention uses the cross-entropy loss function to perform end-to-end training of the model.

[0071] Loss function optimization: The cross-entropy loss function is adopted, treating micro-pose recognition as a single-label classification problem. The formula is as follows:

[0072] in The true label of the i-th sample (one-hot encoding) (This indicates that the i-th sample belongs to the c-th class). This corresponds to the predicted logarithm.

[0073] The cross-entropy loss function combined with AdamW optimizer weight decay can suppress overfitting, improve model generalization accuracy, and ensure that the model converges stably to the optimal solution.

[0074] Step 6: Micro-pose recognition and reasoning.

[0075] After training is complete, the inference phase will proceed as follows: Step 1: Micro-pose video data preprocessing; Step 2: Block embedding and spatiotemporal location coding; Step 3: The bidirectional state-space model module extracts global spatiotemporal features; Step 4: The multi-scale center frame differential state fusion module extracts multi-scale motion sensing features; Step 5: The adaptive scale weighting module dynamically aggregates multi-scale features; Step 6: Global feature aggregation and classification prediction.

[0076] Perform the preprocessing operation in step 1 on the input micro-pose video to obtain a 16-frame video clip; then perform steps 3-6 sequentially to obtain the final fused features. With global feature vectors The predicted logarithm is obtained by inputting it into a linear classifier. The Softmax function maps it to class probabilities.

[0077] The category with the highest probability is selected as the final recognition result, i.e. Simultaneously, the Top-1 accuracy (the percentage of samples whose predicted class matches the true class) and the Top-5 accuracy (the percentage of samples whose top 5 high-probability classes contain the true class) are calculated to evaluate the model performance.

[0078] In the micro-pose recognition inference stage, the process consistent with training preprocessing ensures that the input data distribution matches to avoid a decrease in recognition accuracy. The Softmax function can map the predicted logarithm to probability to intuitively judge the class confidence. Top-1 / Top-5 accuracy can also comprehensively evaluate the model performance and provide direction for subsequent model optimization.

[0079] like Figure 2-3 As shown, Figure 2 The qualitative visualization results of the activation maps output by the method of the present invention under different micro-pose categories show that the present invention explicitly enhances the local spatiotemporal modeling capability of the model through the motion-aware state fusion mechanism, and injects dynamic motion information through the center frame difference operation, enabling the model to accurately capture subtle motion changes in micro-pose and accurately identify multiple micro-pose categories. Figure 3 The visualization results show the attention weights learned by the method of the present invention at different fusion scales under different micro-pose categories. It can be seen that the present invention processes the spatiotemporal context of different receptive fields in parallel and dynamically integrates multi-scale features by using the attention mechanism, so that the model can autonomously select the most suitable feature granularity according to the characteristics of the specific micro-pose, and has the ability to generalize the recognition of micro-poses of different scales and speeds.

[0080] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An efficient micro-pose recognition method based on motion-sensing state fusion Mamba model, characterized in that, Includes the following steps: S1. Data preprocessing: Frame sampling, size normalization, and data augmentation are performed on the micro-pose video. Then, the video frames are converted into spatiotemporal token sequences and embedded with position information. S2, Feature Extraction; For the embedded sequence, a bidirectional state-space model is used to capture global spatiotemporal dependencies and generate global spatiotemporal features of the hidden state sequence. Then, the multi-scale center frame differential state fusion module is used to restore the spatiotemporal structure of the hidden state sequence. The difference between the current frame and the adjacent frames is calculated through the center frame differential operation to explicitly capture motion perception features. The restored global spatiotemporal features and motion perception features are fused at multiple scales, and finally, a multi-scale feature set is output. S3. Multi-scale feature fusion: An adaptive scale weighting module is used to dynamically aggregate multi-scale features to obtain the final fused features. S4. Classification Prediction: The classification prediction module performs global aggregation and linear classification on the final fused features, and outputs the micro-pose category prediction results. S5. The micro-pose recognition model, consisting of a bidirectional state space model, a multi-scale center frame differential state fusion module, an adaptive scale weighting module, and a classification prediction module, is trained and optimized end-to-end using the cross-entropy loss function. S6. Use the trained and optimized micro-pose recognition model for model inference.

2. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 1, characterized in that, In step S1, video frames are converted into spatiotemporal token sequences and embedded with location information, as follows: First, the preprocessed video frames are segmented without overlap using 3D convolutional kernels, converting them into a spatiotemporal patch sequence. Then, each spatiotemporal patch is mapped to a feature token using a linear layer, resulting in a spatiotemporal token sequence. This is followed by a sequence of spacetime tokens. Add learnable spatial embeddings and time location embedding The final embedded sequence is This sequence preserves both spatiotemporal features and location information.

3. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 1, characterized in that, The bidirectional state-space model in S2 is as follows: The bidirectional state space model includes two parallel branches: a forward state space model and a reverse state space model. Both the forward and reverse state space models adopt a discrete state space model structure. Both the forward state-space model and the reverse state-space model contain independent parameters, with the core parameter being the state transition matrix. Input matrix Output matrix ,in N To hide the state dimension; The continuous state-space model is discretized using the zero-order preservation method, resulting in the discrete matrix: ,in Let I be the discretization step size, and let I be the identity matrix.

4. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 3, characterized in that, The processing procedure for the bidirectional state-space model is as follows: First, bidirectional feature calculation: The forward state-space model inputs the embedded sequence Z in its original time order and updates it according to the discrete state update formula. , For the first Time tokens generate positive hidden states The inverse state-space model is obtained by performing time flipping on the embedded sequence Z. Generate reverse hidden state And then Flip to get ; Subsequent bidirectional feature fusion: fusion through averaging operations To obtain global spatiotemporal features .

5. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 1, characterized in that, The multi-scale center frame differential state fusion module in S2 includes state reshaping operation, center frame differential operation, multi-scale state fusion mechanism and feature synthesis mechanism; The state reshaping operation reshapes a one-dimensional hidden state sequence into a structured four-dimensional tensor, which serves as the original feature. The center frame differential operation explicitly calculates the dynamic changes between adjacent frames to generate motion-aware features; The multi-scale state fusion mechanism applies a multi-scale state fusion operator to locally weight and aggregate the original features and motion-sensing features in 3D neighborhoods of different sizes. The feature synthesis mechanism achieves adaptive fusion of original features and motion-sensing features through learnable gating parameters.

6. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to any one of claims 1-5, characterized in that, The processing procedure of the multi-scale center frame differential state fusion module is as follows: First, feature dimension reshaping: the global spatiotemporal features output by the bidirectional state-space model. Remodeling into 4D tensors Recovery time T and space Dimensional structure; Next, the center frame difference is calculated: the center frame difference is calculated in the time dimension. : in, Finally, motion perception features are obtained. ; Subsequently, multi-scale local fusion is implemented: three parallel scale branches are set up, and each branch processes F and D through a local weighted aggregation operation. in, For 3D neighborhood, Corresponding window radii of 1, 2, and 3, Learnable kernel weights; Finally, feature fusion and output: Introducing learnable gating parameters. Integrating original features with motion-sensing features: This yields a multi-scale feature set.

7. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 6, characterized in that, The adaptive scale weighting module in S3 includes multi-scale feature splicing and stacking operations, a lightweight 3D convolutional attention subnetwork, and weighted summation operations.

8. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 7, characterized in that, The processing procedure of the adaptive scale weighting module is as follows: Feature concatenation: Concatenates multi-scale features along the channel dimension to obtain a unified feature. ,Will Stacked as ; Attention weight calculation: based on unified features The scale attention weights are generated using a two-layer 3D convolutional attention subnetwork, and then obtained by softmax normalization. ; Weighted fusion: Through broadcasting mechanism and attention weight Sum along the scale dimension after multiplication: Obtain the final fusion features .

9. The efficient micro-pose recognition method based on motion-aware state fusion Mamba model according to claim 8, characterized in that, The classification prediction module in S4 is as follows: The classification prediction module includes a global aggregation module and a linear classifier; The final fused features are flattened and global average pooled in the global aggregation module to aggregate global information and obtain a global feature vector. The global feature vector is input into a linear classifier to obtain the prediction results of the micro-pose category.

10. The efficient micro-pose recognition system based on the motion-aware state fusion Mamba model obtained by the method described in claim 9, characterized in that, include: The data preprocessing module performs frame sampling on the micro-attitude video to be tested, and converts the video frames into a spatiotemporal token sequence and embeds position information. The micro-pose recognition model includes a bidirectional state space model, a multi-scale center frame differential state fusion module, an adaptive scale weighting module, and a classification prediction module. The output selects the category with the highest prediction probability as the final micro-pose recognition result of the micro-pose video to be tested.

Images