A four-stream human action recognition method based on multi-modal hierarchical adaptive fusion

Abstract

This invention discloses a four-stream human action recognition method based on multimodal hierarchical adaptive fusion, comprising: acquiring video skeleton data to be recognized; preprocessing the video skeleton data; inputting the preprocessed data into a trained four-stream human action recognition model to obtain four-stream human action recognition results; wherein the four-stream human action recognition model includes a four-stream backbone network, a four-stage progressive fusion module, a temporal dynamic weight generation module, a temporal pooling module, and a classification module; this invention, through a cognitive bionic four-stage hierarchical adaptive fusion architecture, has significant advantages over existing technologies in terms of multimodal information utilization, feature discriminability, model robustness, and computational efficiency, providing strong technical support for the efficient and accurate application of human action recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, and specifically relates to a four-stream human action recognition method based on multimodal hierarchical adaptive fusion. Background Technology

[0002] Human action recognition, as an important research direction in computer vision and artificial intelligence, is widely used in intelligent security monitoring, human-computer interaction, motion analysis, virtual reality, and medical rehabilitation assistance. Traditional methods are mainly based on handcrafted features and traditional machine learning algorithms. However, these methods are limited by their feature representation capabilities and struggle to handle complex scenarios involving changes in action, occlusion, and viewpoint. In recent years, with the development of deep learning technology, especially breakthroughs in convolutional neural networks (CNNs) and graph convolutional networks (GCNs), significant progress has been made in human action recognition. Existing methods can be mainly divided into the following three categories:

[0003] (1) Monomodal methods based on RGB video: 2D / 3D CNNs are used to extract spatiotemporal features from RGB videos, such as I3D, SlowFast, and R(2+1)D. These methods rely solely on visual information, are sensitive to changes in illumination and background noise, and struggle to capture the structured information of the human body, easily mistaking background noise for action features. Recent research, such as VideoMAE V2, further enhances visual representation capabilities through self-supervised pre-training, but still cannot solve the inherent defect of being sensitive to background noise. (2) Monomodal methods based on skeleton data: GCNs are used to model human joint sequence, such as ST-GCN, CTR-GCN, and MS-G3D. These methods utilize prior knowledge of human structure, are naturally robust to changes in illumination and background clutter, and can accurately describe human posture, but lack appearance information, making it difficult to distinguish visually similar actions and perceive the interaction between humans and objects. (3) Given the limitations of single-modal information, multimodal action recognition methods that integrate visual features and skeleton structure features have become a current research hotspot, such as MMNet and PoseConv3D.

[0004] Existing methods based on RGB video include the SlowFast network, which uses a dual-stream architecture to process RGB video, capturing spatial details through a slow, high-resolution path and temporal dynamics through a fast, low-resolution path. However, this method is limited to spatiotemporal modeling within the visual modality, lacking explicit representation of human body topology and susceptible to background noise. Single-modal methods based on skeleton data include CTR-GCN, which performs skeleton action recognition through channel-level topology refinement map convolution, offering high accuracy but lacking appearance information and not considering multimodal fusion. MS-G3D, which performs skeleton action recognition through multi-scale spatiotemporal map convolution, can capture long-distance joint dependencies, but is also limited to the skeleton modality.

[0005] The existing technologies described above have the following problems: 1) Incomplete single-modal information: Methods relying solely on RGB are limited by environmental factors (lighting, background) and have poor robustness; methods relying solely on skeletons lack appearance and texture information, resulting in low discrimination of "interactive" actions (such as reading and writing) and limited recognition accuracy. 2) Shallow multimodal fusion mechanism: Existing fusion methods are mostly feature splicing or post-decision fusion, lacking in-depth modeling of complex nonlinear interaction relationships between modalities, and failing to fully utilize the deep complementarity between visual flow and skeleton flow, resulting in limited improvement in fusion effect. 3) Lack of modal decoupling design and feature redundancy: Existing fusion architectures do not consider the distribution differences and primary and secondary relationships of different modalities in the feature space. Direct fusion often leads to high-noise modalities (such as RGB affected by background interference) contaminating low-noise modalities (such as pure skeleton features), causing mutual interference and feature redundancy between modalities, which in turn reduces the model's discriminative ability. 4) High computational complexity and difficult deployment: Existing methods often simply stack networks when processing multimodal data, which leads to a significant increase in the number of parameters and computational load, making it difficult to deploy in embedded devices or real-time systems with limited computing resources. Summary of the Invention

[0006] To address the issues of effective fusion of multimodal information, modal decoupling, feature redundancy, and computational efficiency in existing technologies, this invention proposes a four-stream human action recognition method based on multimodal hierarchical adaptive fusion. The method includes: acquiring multimodal data of the sample to be recognized, wherein the multimodal data, after preprocessing, constitutes four-stream data: RGB spatial stream, optical flow temporal stream, skeleton structure stream, and skeleton motion stream; inputting the four-stream data into a trained four-stream human action recognition model to obtain the four-stream human action recognition result; wherein the four-stream human action recognition model includes a four-stream backbone network, a four-stage progressive fusion module, a temporal dynamic weight generation module, a temporal pooling module, and a classification module.

[0007] Training the four-stream human motion recognition model includes: acquiring a training dataset; preprocessing the data in the training dataset to obtain four-stream training data; inputting the four-stream data into a four-stream backbone network for feature extraction to obtain four-stream features, which include: spatial semantic features, temporal dynamic features, static pose topology features of human joint sequence, and dynamic motion features of human joint sequence; using a four-stage progressive fusion module to perform deep interactive aggregation of the four-stream features to obtain interactive features; using a temporal dynamic weight generation module to fuse the interactive features to obtain fused features; using a temporal pooling module to perform temporal pooling processing on the fused features; inputting the temporally pooled features into a classifier to obtain recognition results; constructing the model's loss function based on the recognition results; using the AdamW optimizer to optimize the model parameters; and completing model training when the preset number of training rounds is reached or the preset training termination condition is met.

[0008] The beneficial effects of this invention are:

[0009] 1. This invention simulates the human cognitive process of "local to global" through a four-stage hierarchical fusion architecture, enabling hierarchical deep interaction of four-stream features; intramodal interaction strengthens the internal relationships of similar modalities and eliminates internal modal noise; cross-modal interaction establishes clear physical attribute correspondences; supplementary interaction enhances cross-modal understanding and uncovers higher-order implicit associations; global interaction achieves the final coordination and conflict resolution of four-stream information; hierarchical residual connections construct a progressive residual connection chain between each interaction layer, with the output of each stage added to the input of the previous stage, forming an information highway that runs through the four layers; this design ensures that the original information is transmitted to the deep layers without loss, while making the fusion process naturally interpretable—by analyzing the attention distribution of each stage, it is possible to observe how the model gradually understands the actions, and this interpretability provides an intuitive basis for model debugging and error analysis.

[0010] 2. To address the computational explosion problem caused by deep multimodal fusion, this invention innovatively proposes a symmetric cross-attention mechanism. This mechanism, by introducing parameter sharing and flow identifier embedding, merges the traditional bidirectional attention, which requires two independent computations, into a single self-attention computation, simultaneously achieving intra-flow self-enhancement and inter-flow cross-modal interaction. Theoretical analysis and experimental verification show that this mechanism, while maintaining equivalent representational capabilities, reduces redundant parameters and repetitive computation overhead, effectively solving the zero-sum game of "performance-efficiency" in multimodal fusion. It achieves a unification of deep semantic interaction and lightweight computation, providing key technical support for the deployment of models in embedded devices or real-time monitoring systems.

[0011] 3. To address the modal competition and noise contamination issues caused by the bidirectional convergence strategy in existing multimodal alignment methods, this invention proposes a unidirectional gradient decoupling cross-modal alignment strategy. In structure-space alignment and motion-temporal alignment, a gradient detach operation is performed on visual modal features to block their gradient backpropagation path, retaining only the gradient updates of skeleton modal features. This design ensures that the skeleton modality actively learns the semantic distribution of the visual modality as the core anchor point, while isolating visual noise from contaminating the skeleton feature extractor. Feature visualization analysis shows that this strategy amplifies inter-class differences and reduces intra-class differences in fused features, decreases feature redundancy, and significantly improves the model's discriminative ability in complex environments.

[0012] This invention, through a cognitive bionic four-stage hierarchical adaptive fusion architecture, has significant advantages over existing technologies in terms of multimodal information utilization, feature discriminability, model robustness, and computational efficiency, providing strong technical support for the efficient and accurate application of human action recognition. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating the overall network flow framework of the present invention.

[0014] Figure 2 This is a diagram showing the internal structure of the four-stage progressive fusion module of the present invention.

[0015] Figure 3 This is a schematic diagram of the symmetrical cross-attention mechanism of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] A four-stream human action recognition method based on multimodal hierarchical adaptive fusion, such as Figure 1 As shown, the method includes: acquiring multimodal data of the sample to be identified, wherein the multimodal data is preprocessed to form four-stream data: RGB spatial stream, optical flow temporal stream, skeleton structure stream, and skeleton motion stream; inputting the preprocessed data into a trained four-stream human action recognition model to obtain four-stream human action recognition results; wherein the four-stream human action recognition model includes a four-stream backbone network, a four-stage progressive fusion module, a temporal dynamic weight generation module, a temporal pooling module, and a classification module.

[0018] Training the four-stream human motion recognition model includes: acquiring a training dataset; preprocessing the data in the training dataset to obtain four-stream data; inputting the four-stream data into a four-stream backbone network for feature extraction to obtain four-stream features, which include: spatial semantic features, temporal dynamic features, static pose topology features of human joint sequence, and dynamic motion features of human joint sequence; using a four-stage progressive fusion module to perform deep interactive aggregation of the four-stream features to obtain interactive features; using a temporal dynamic weight generation module to fuse the interactive features to obtain fused features; using a temporal pooling module to perform temporal pooling processing on the fused features; inputting the temporally pooled features into a classifier to obtain recognition results; constructing the model's loss function based on the recognition results; using the AdamW optimizer to optimize the model parameters; and completing model training when the preset number of training rounds is reached or the preset training termination condition is met.

[0019] In this embodiment, the video skeleton data includes RGB video and 3D skeleton data; the RGB video includes RGB spatial stream and optical flow temporal stream; the 3D skeleton data includes skeleton structure stream and skeleton motion stream; missing skeleton data and poor quality data are removed from the video skeleton data to obtain the sample to be processed; the sample to be processed is uniformly regularized based on the non-equidistant keyframe sampling strategy to obtain the alignment sequence; the alignment sequence is normalized to obtain the four-stream data.

[0020] The non-uniformly spaced keyframe sampling strategy unifies and normalizes the samples to be processed, including: constructing temporal energy curves based on video modality and skeleton modality respectively, and calculating their gradient change sequences; calculating dynamic modality weights based on the gradient change sequences; establishing a unified temporal mapping relationship between video frames and skeleton frames based on dynamic modality weights and global linear alignment or multi-scale cross-correlation alignment; detecting peak anchors, turning anchors, and boundary anchors within the effective action interval, and constructing a joint score for candidate frames based on peak anchors, turning anchors, and boundary anchors; using a diversity sampling method with minimum interval constraints to non-uniformly select keyframes from a unified time axis; and completing or cropping the keyframes based on the unified temporal mapping relationship to output a strongly aligned sequence that is strictly normalized to T=16 frames.

[0021] In this embodiment, a diversity sampling method with minimum interval constraints is adopted to non-uniformly select keyframes from a unified time axis, including: acquiring frame sample data and normalizing the features of all samples; calculating the pairwise distance matrix between samples; randomly selecting the first sample from the entire set X and placing it into the result set S, and using the samples in the remaining sample set U as candidate samples; calculating the minimum distance from each candidate sample u to the existing set S, and when the minimum distance is greater than a preset threshold, the sample is taken as a candidate set; selecting the sample with the largest minimum distance from the candidate set; adding the sample to S and removing it from U; outputting samples in real time when a specified number k samples are collected; traversing all pairwise samples in the final S, and if there are samples with a distance less than the threshold, removing conflicting samples and adding new compliant samples; finally, outputting the globally diverse optimal sampling set, the data in this set being the keyframes.

[0022] In this embodiment, the four-stream backbone feature extraction module is used to extract raw features from four modalities of data; and to unify the output dimension [B, 16, 256] of the features extracted by the four parallel backbone networks; specifically including:

[0023] RGB Spatial Stream: Using the ConvNeXt-V2 backbone, it extracts the spatial semantic features of RGB images and outputs [B, 16, 256].

[0024] RGB Temporal Stream: Using an R(2+1)D backbone, extract the dynamic features of optical flow temporal stream from RGB video and output [B,16,256].

[0025] Structural Stream: Using the CTR-GCN backbone, it extracts the static pose topology features of the human joint sequence and outputs [B,16,256].

[0026] Skeleton Motion Stream: Using the MS-G3D backbone, extracts the dynamic motion features of human joint sequence and outputs [B, 16, 256].

[0027] In this embodiment, the four-stage progressive fusion module includes: an intramodal interaction layer, a same-attribute cross-modal interaction layer, a heterogeneous attribute supplementary interaction layer, and a global adaptive fusion layer; the intramodal interaction layer includes a skeleton intramodal interaction unit and a visual intramodal interaction unit; the same-attribute cross-modal interaction layer includes a static attribute interaction unit and a dynamic attribute interaction unit; the heterogeneous attribute supplementary interaction layer includes a structure-time interaction unit and a motion-space interaction unit.

[0028] The hierarchical adaptive fusion module is used for deep interactive fusion of four-stream features. It includes four progressive stages, each introducing residual connections to preserve the original information. The specific structure is as follows: Figure 2 As shown.

[0029] Phase 1: Intramodal Interaction Layer. Purpose: This layer enhances the complementarity within homogeneous modes and eliminates intramodal noise before encountering heterogeneous information. It includes two parallel units:

[0030] Intramodal interaction unit of skeleton: performs bidirectional cross-attention interaction between skeleton structure flow F_s and skeleton motion flow F_m, and outputs F_s_intra and F_m_intra;

[0031] Visual modal interaction unit: performs bidirectional cross-attention interaction between visual spatial flow F_rs and visual temporal flow F_rt, and outputs F_rs_intra and F_rt_intra;

[0032] Residual connection: Each output is added to its corresponding input to form the first layer of enhanced features.

[0033] Phase Two: Cross-Modal Interaction (CMO) Purpose: This layer establishes cross-modal correspondences between human pose and visual appearance, and between skeletal motion and video optical flow, based on physical attribute pairing. It includes two parallel units:

[0034] Static attribute interaction unit: The symmetric cross attention mechanism is used to interact the skeleton structure flow F_s_intra and the visual space flow F_rs_intra of the first layer to establish the correspondence between static pose and static appearance, and output F_s_cross and F_rs_cross.

[0035] Dynamic attribute interaction unit: The symmetrical cross attention mechanism is used to interact the skeleton motion flow F_m_intra and the visual temporal flow F_rt_intra of the first layer, establish the correspondence between dynamic motion and dynamic temporal sequence, and output F_m_cross and F_rt_cross.

[0036] Residual connection: Each output is added to the corresponding input of the first layer to form the enhanced features of the second layer.

[0037] Phase 3: Supplementary Interaction. Purpose: This layer is used to mine higher-order implicit associations, capturing the "influence of pose on velocity" and the "dependence of motion on background," enhancing cross-modal understanding and filling semantic blind spots. It includes two parallel units:

[0038] Structure-Time Interaction Unit: Interacts with the second-layer skeleton structure flow F_s_cross and the visual temporal flow F_rt_cross to capture the "influence of static pose on dynamic timing" and outputs F_s_supp and F_rt_supp;

[0039] Motion-space interaction unit: Interacts with the skeleton motion flow F_m_cross and the visual spatial flow F_rs_cross of the second layer, captures the "dependence of dynamic motion on static background", and outputs F_m_supp and F_rs_supp;

[0040] Residual connection: Each output is added to the corresponding input of the second layer to form the enhanced features of the third layer.

[0041] Phase 4: Global Adaptive Fusion. Purpose: This layer is used to achieve global coordination and conflict resolution of the four-stream information. The four-stream features output from layer 3 are stacked into a joint tensor of [B, 4, N, D]. After adding stream identifier embeddings, it is reshaped into a sequence of [B, 4N, D]. Efficient global attention is used to perform a one-time self-attention calculation, processing all frames of all streams simultaneously, capturing the global dependencies between the four streams. The output is then joined by residual connections (added to the output of layer 3) to form the enhanced features of layer 4.

[0042] In this embodiment, the temporal dynamic weight generation module includes a feature aggregation subunit, a temporal attention subunit, a weight generation subunit, and a weighted fusion subunit. This module is connected to four interaction layers and is used to perform temporally-aware adaptive hierarchical fusion of the four-layer historical features of each stream. The temporal dynamic weight generation module processes the interaction features as follows: the feature aggregation subunit concatenates the four-layer features of each stream along the channel dimension to form a hierarchical feature sequence; the temporal attention subunit interacts with the hierarchical feature sequence to evaluate the importance of the four-layer features at each time step; based on the importance of the four-layer features, the weight generation subunit generates four-layer fusion weights for each time step, and normalizes them using Softmax to output a [B,N,4] temporal dynamic weight matrix; based on the generated temporal dynamic weights, the weighted fusion subunit performs a weighted summation of the four-layer features to output the final enhanced features of each stream.

[0043] This design enables the fusion process to be time-sensitive and hierarchically adaptive, dynamically adjusting the contribution of features at each layer according to different stages of the action, achieving finer fusion control than global fixed weights.

[0044] In this embodiment, the temporal pooling and classification output module processes the data by performing temporal pooling and classification on the fused features and outputting action category prediction results, including multi-scale temporal pooling (global attention + local convolution + temporal attention), progressive classification network (256→128→64→32→60), and an auxiliary classifier to provide intermediate supervision.

[0045] The four-stage progressive fusion architecture of this invention—intramodal → cross-modal with similar attributes → heterogeneous attribute supplementation → global—is irreplaceable. This design simulates the human cognitive process: first, understanding the internal relationships of a single modality (seeing the action and posture); then, establishing cross-modal correspondences with similar attributes (connecting with the environmental context); next, engaging in heterogeneous attribute supplementation interaction (understanding from multiple perspectives); and finally, performing global integration (comprehensive judgment). Other simple splicing or single-layer attention cannot achieve this hierarchical cognitive simulation process, and this architecture provides a structural foundation for deep multimodal fusion.

[0046] The core computational unit used in this invention is a symmetric cross-attention mechanism, such as... Figure 3 As shown, this mechanism achieves bidirectional information interaction and self-reinforcement through stream identifier embedding and parameter sharing, symmetric partitioning, and residual connections, enabling bidirectional information interaction and self-reinforcement between any two streams with a single self-attention computation. This mechanism is reused in all interaction layers of the fusion module of this invention, specifically including: intra-modal interaction (S↔M)(RS↔RT); inter-modal interaction (S↔RS)(M↔RT); and supplementary interaction (S↔RT)(M↔RS). Each unit shares the same computational logic, achieving parameter unification and computational efficiency, thus solving the problem of computational explosion caused by multimodal deep fusion.

[0047] This invention constructs a progressive residual connection chain at each interaction stage. The output of each stage consists of two parts: the original interaction output of the current stage and the final output of the previous stage, forming a hierarchical feature transfer path. It has the following technical effects: Lossless information transfer: The original input can be directly transferred to any subsequent stage through residual connections. Even after multiple layers of interaction, the underlying details are still fully preserved, preventing deep network degradation; Smooth gradient propagation: During backpropagation, the gradient of the loss function can flow directly to the lower layers through residual connections, alleviating the gradient vanishing problem in deep networks and ensuring that the parameters of all four stages are fully optimized; Hierarchical feature accumulation: The output of each stage retains the results of all previous stages and superimposes the new understanding of the current stage, forming a rich feature hierarchy; Decoupling optimization: The residual branch of each stage only needs to learn the residual relative to the previous stage, reducing the learning difficulty of each subtask and making training more stable and efficient.

[0048] This invention introduces a unidirectional gradient decoupling mechanism at the training strategy level. This mechanism is an innovation in training strategies and, together with the aforementioned architectural innovation (four-stage fusion) and mechanism innovation (symmetric attention), constitutes a complete technical solution. Specifically, in the calculation of structure-space alignment loss and motion-temporal alignment loss, a gradient detach operation is performed on the visual modality features. This design is fundamentally different from the existing bidirectional alignment strategies: existing technologies simultaneously bring the feature distributions of two modalities closer together, and the gradient updates the feature extractors of both modalities simultaneously, leading to visual noise contaminating the skeleton features. This invention uses only the skeleton modality as an anchor point, with the visual modality acting as a "teacher" to provide semantic distribution targets but not participating in gradient updates. This ensures a unidirectional flow of skeleton information to the visual modality, avoids visual noise contaminating the skeleton features, and realizes the design concept of "skeleton-led, vision-assisted."

[0049] In this embodiment, NTU RGB+D 60 human motion recognition based on four-stream hierarchical fusion includes:

[0050] Step 1: Dataset preparation and preprocessing. This includes:

[0051] Step 1.1: Obtain the NTU RGB+D 60 dataset, which contains 60 action classes and a total of 56,880 video samples. Each sample contains RGB video, depth map, 3D skeleton and infrared video.

[0052] Step 1.2: Specific preprocessing of the four-stream data. Preprocessing is performed on the RGB video and 3D skeleton data to remove missing skeleton data and data with severely insufficient quality. Based on a non-uniformly spaced keyframe sampling strategy, heterogeneous video frames and skeleton sequences are uniformly normalized into a strongly aligned sequence of T=16 frames. This preserves the key temporal evolution of the motion and solves the problem of batch processing inability to process variable-length sequences. The specific four-stream data is obtained through each stream processor. In this embodiment, RGB video, optical flow sequence, skeleton structure sequence, and skeleton motion sequence constitute the four-stream input.

[0053] Step 1.3: According to the official dataset protocol, Cross-Subject (X-Sub) is divided by person identity, with 40,320 training samples and 16,560 test samples;

[0054] Cross-View (X-View): Divided by camera viewpoint, 37,920 training samples and 18,960 test samples.

[0055] Step 2: Build a four-stream backbone network. The four-stream backbone network includes the ConvNeXt-V2 backbone, the R(2+1)D backbone, the CTR-GCN backbone, and the MS-G3D backbone.

[0056] Step 2.1: Extract RGB video frame spatial semantic features using the pre-trained ConvNeXt-V2 model. The specific processing is as follows: Expand the input tensor [B,T,3,H,W] along the time dimension into a set of frame-by-frame images, transforming it into [B×T,3,H,W]; input the frame-by-frame images into the ConvNeXt-V2 backbone to extract high-dimensional spatial features, obtaining frame-by-frame visual features; restore the frame-by-frame visual features to the temporal form [B,T,C rs], where C rs is the backbone output channel; map the backbone output to the common feature dimension D through a fully connected projection layer, obtaining [B, T, D]; input the projected features into the temporal refinement module, which preferably includes one-dimensional convolution, batch normalization, activation function, and one-dimensional convolution, enhancing local temporal dependencies through temporal convolution; add the temporal refinement output and the projected features by residual addition to obtain the final RGB spatial flow features, with an output dimension of [B, 16, D].

[0057] Preferably, the value of the common feature dimension D is 256.

[0058] Step 2.2: Extract the optical flow temporal dynamic features of RGB video using the R(2+1)D backbone. The specific process is as follows: rearrange the input from [B,T,2,H,W] into a form suitable for 3D convolution [B,2,T,H,W]; use the R(2+1)D backbone network to extract spatiotemporal features from the input, and perform global average pooling on the backbone output only in the spatial dimension, retaining the temporal dimension information to obtain [B, C rt,T′]; transpose it to [B,T′,C rt], and map it to the common dimension 256 through a fully connected projection layer; input the aligned features into the temporal refinement module, and output the final optical flow temporal features with dimensions [B,16,256].

[0059] Step 2.3: Extract static pose topological features of human joint sequence using the CTR-GCN backbone; the CTR-GCN backbone includes 10 CTRGCNBlocks; the number of channels is [64, 64, 64, 64, 128, 128, 128, 256, 256, 256], and the input skeleton structure flow tensor is [B, C, T, V]. The specific process is as follows: Construct a human skeleton graph structure, representing joints and their topological connections as an adjacency matrix; feed the input skeleton structure sequence into the CTR-GCN backbone, and extract structural features through alternating multi-layer graph convolution and temporal convolution; gradually increase the feature dimension from low dimension to high dimension during the channel enhancement stage to enhance representation capability; introduce a spatial attention module at the end of the backbone to enhance the response of key joints; aggregate the output features in the joint dimension and rearrange them into a unified temporal format to obtain skeleton structure flow features.

[0060] Preferably, the size of the output skeleton structure flow feature is [B, 16, 256].

[0061] Step 2.4: Use the MS-G3D backbone to extract dynamic motion features of the human joint sequence; the MS-G3D backbone contains 10 MSG3DBlocks with [96, 96, 96, 96, 192, 192, 192, 384, 384, 384] channels; the specific process is as follows: normalize and format the skeleton motion sequence; feed the input into the MS-G3D backbone for spatiotemporal graph convolution modeling; preferably, a multi-stage parallel structure is adopted inside the backbone to jointly model local motion patterns and long-range spatiotemporal dependencies; map the output high-dimensional features to a common dimension of 256 through a projection layer; aggregate the joint dimensions and rearrange them into a unified temporal format to obtain the skeleton motion flow features.

[0062] Preferably, the size of the output skeleton motion flow feature is [B, 16, 256].

[0063] Step 3: Build a four-stage progressive integration module.

[0064] Step 3.1: Construct the core module for Symmetric CrossAttention. To implement SymmetricCrossAttention classification, the following process is included:

[0065] Stream label embedding: Initialize learnable label vectors E_A and E_B for the two input streams respectively, with dimensions [1,1,256];

[0066] Joint multi-head attention: Multi-head attention is used to model the self-attention of the concatenated joint sequence;

[0067] Segmentation: The processed joint sequence is segmented according to its original length;

[0068] Residuals and Normalization: Perform Dropout, residual summation, and LayerNorm on the attention output;

[0069] Gated fusion: Gated residual fusion is performed on the segmented dual-stream enhanced features to control the intensity of the enhanced information injection.

[0070] During forward propagation: Input A, B → Add labels → Concatenate → Joint multi-head attention → Segment → Add to the original input → Layer normalization → Output enhanced A, B.

[0071] Step 3.2: Construct a four-stage fusion network. The data is processed through a four-stage fusion network, including: Stage 1: Two SymmetricCrossAttention mechanisms process skeleton pairs and visual pairs respectively; Stage 2: Two SymmetricCrossAttention mechanisms process static pairs and dynamic pairs respectively; Stage 3: Two SymmetricCrossAttention mechanisms process structure-time and motion-space; Stage 4: A global self-attention mechanism processes all four stream features.

[0072] Step 3.3: Implement the temporal dynamic weight generation module. This includes: Feature aggregation: Layer stacking, concatenating the four-layer output of each stream [B, 16, 256×4]. Temporal attention: Attention / MLP + Sigmoid to generate weights of [B, 16, 4]. Weighted fusion: Weighted summation of features from each layer, outputting the final enhanced features for each stream.

[0073] Step 4: Training configuration.

[0074] Step 4.1: Loss Function Configuration. This invention employs multi-task loss collaborative optimization, with the specific configuration as follows: The final classification logits output by the fusion module are denoted as... Where B is the batch size, C is the number of action categories, and y is the true label; the expression for the loss function is:

[0075]

[0076] Main classification loss: Used to supervise the final classification output of the fused features, enabling the model to learn the mapping relationship between the global fused representation and the action category; a labeled smooth cross-entropy loss is employed, where... This represents the labeled smooth cross-entropy. The primary classification loss weight.

[0077] Auxiliary classification loss: To enhance the discriminative ability of each sub-stream branch and avoid relying solely on the final fusion result during model training while ignoring single-stream feature learning; it is defined as the average of the cross-entropy losses of the four branches, and the logits of the auxiliary classification head outputs of the four single-stream branches are denoted as: ,in To assist in loss weighting.

[0078] Feature regularization term: To prevent abnormally large feature amplitudes, unstable distributions, and overfitting during model training; defined as the average of the squared means of the feature elements in each group, where... This represents the arithmetic mean of the squares of all elements of the characteristic tensor. The weights are the feature regularization terms.

[0079] Step 4.2: Optimizer Configuration. The optimizer used is the AdamW optimizer, with an initial learning rate of 1e-3; weight decay of 1e-4; learning rate scheduling of cosine annealing with warmup restart; 3 warmup rounds; 300 training rounds; and a batch size of 16.

[0080] Step 4.3: Data Augmentation. Augmentation of RGB frames: random horizontal flipping, random cropping, random scaling, color dithering, random erasing; Skeleton flow augmentation: random rotation (±30°), random scaling (0.9-1.1), random Gaussian noise, joint occlusion, bone occlusion; Temporal augmentation: temporal warp.

[0081] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A four-stream human action recognition method based on multi-modal hierarchical adaptive fusion, characterized in that, include: The multimodal data of the sample to be identified is acquired. After preprocessing, the multimodal data is composed of four streams: RGB spatial stream, optical flow temporal stream, skeleton structure stream, and skeleton motion stream. The four-stream data is input into the trained four-stream human action recognition model to obtain the four-stream human action recognition results; the four-stream human action recognition model includes a four-stream backbone network, a four-stage progressive fusion module, a temporal dynamic weight generation module, a temporal pooling module, and a classification module. Training the four-stream human motion recognition model includes: acquiring a training dataset; preprocessing the data in the training dataset to obtain four-stream data; inputting the four-stream data into a four-stream backbone network for feature extraction to obtain four-stream features, which include: spatial semantic features, temporal dynamic features, static pose topology features of human joint sequence, and dynamic motion features of human joint sequence; using a four-stage progressive fusion module to perform deep interactive aggregation of the four-stream features to obtain interactive features; using a temporal dynamic weight generation module to fuse the interactive features to obtain fused features; using a temporal pooling module to perform temporal pooling processing on the fused features; inputting the temporally pooled features into a classifier to obtain recognition results; constructing a loss function for the model based on the recognition results; using the AdamW optimizer to optimize the model parameters; and completing the model training when the loss function converges.

2. The four-stream human action recognition method based on multi-modal hierarchical adaptive fusion according to claim 1, characterized in that, Video skeleton data includes RGB video and 3D skeleton data; RGB video includes RGB spatial stream and optical flow temporal stream; 3D skeleton data includes skeleton structure stream and skeleton motion stream. Remove missing skeletons and poor-quality data from the video skeleton data to obtain the sample to be processed; standardize the sample to be processed based on the non-equidistant keyframe sampling strategy to obtain the alignment sequence; normalize the alignment sequence to obtain the four-stream data.

3. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 2, characterized in that, The non-uniformly spaced keyframe sampling strategy is used to uniformly normalize the samples to be processed, including: constructing temporal energy curves based on video modality and skeleton modality respectively, and calculating gradient change sequences of RGB video and 3D skeleton data; calculating dynamic modality weights based on gradient change sequences; establishing a unified temporal mapping relationship between video frames and skeleton frames based on dynamic modality weights and global linear alignment or multi-scale cross-correlation alignment; detecting peak anchors, turning anchors, and boundary anchors within the effective action interval, and constructing a joint score for candidate frames based on peak anchors, turning anchors, and boundary anchors; using a diversity sampling method with minimum interval constraints to non-uniformly select keyframes from a unified time axis; and completing or cropping the keyframes based on the unified temporal mapping relationship to output a strongly aligned sequence of T=16 frames.

4. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 1, characterized in that, The four-stream backbone network includes the ConvNeXt-V2 backbone, the R(2+1)D backbone, the CTR-GCN backbone, and the MS-G3D backbone; The ConvNeXt-V2 backbone is used to extract spatial semantic features of RGB video frames; the R(2+1)D backbone is used to extract the optical flow temporal dynamic features of RGB video; and the CTR-GCN backbone is used to extract the static pose topology features of human joint sequence. Using the MS-G3D backbone, dynamic motion features of human joint sequence are extracted.

5. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 1, characterized in that, The four-stage progressive fusion module includes: an intramodal interaction layer, a same-attribute cross-modal interaction layer, a heterogeneous attribute supplementary interaction layer, and a global adaptive fusion layer; the intramodal interaction layer includes a skeleton intramodal interaction unit and a visual intramodal interaction unit; the same-attribute cross-modal interaction layer includes a static attribute interaction unit and a dynamic attribute interaction unit; the heterogeneous attribute supplementary interaction layer includes a structure-time interaction unit and a motion-space interaction unit.

6. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 4, characterized in that, The four-stage progressive fusion module performs deep interactive fusion of features from the four flows, including: In the first stage, within the intramodal interaction layer, the skeleton structure flow F_s and the skeleton motion flow F_m are input into the skeleton intramodal interaction unit for bidirectional cross-attention interaction, resulting in F_s_intra and F_m_intra; the visual spatial flow F_rs and the visual temporal flow F_rt are input into the visual intramodal interaction unit for bidirectional cross-attention interaction, resulting in F_rs_intra and F_rt_intra; each output is added to its corresponding input to form the first layer of enhanced features; In the second stage, in the same-attribute cross-modal interaction layer, the cross-attention mechanism of the static attribute interaction unit is used to interact the skeleton structure flow F_s_intra and the visual spatial flow F_rs_intra of the first layer, establishing the correspondence between static pose and static appearance, and outputting F_s_cross and F_rs_cross; the symmetric cross-attention mechanism of the dynamic attribute interaction unit is used to interact the skeleton motion flow F_m_intra and the visual temporal flow F_rt_intra of the first layer, establishing the correspondence between dynamic motion and dynamic temporal sequence, and outputting F_m_cross and F_rt_cross; each output is added to the corresponding input of the first layer to form the second layer of enhanced features; In the third stage, in the heterogeneous attribute supplementation interaction layer, a structure-temporal interaction unit is used to interact with the skeleton structure flow F_s_cross and the visual temporal flow F_rt_cross of the second layer to capture the influence of static pose on dynamic temporal sequence and output F_s_supp and F_rt_supp; a motion-space interaction unit is used to interact with the skeleton motion flow F_m_cross and the visual spatial flow F_rs_cross of the second layer to capture the dependence of dynamic motion on static background and output F_m_supp and F_rs_supp; each output is added to the corresponding input of the second layer to form the third layer of enhanced features; In the fourth stage, in the global adaptive fusion layer, the four-stream features output from the third layer are stacked along the stream dimension and reshaped into a sequence of [B, 4N, D] after adding stream identifier embedding. Efficient global attention is used to perform a one-time self-attention calculation to process all frames of all streams simultaneously, capturing the global dependencies between the four streams. The global dependencies are added to the output of the third layer to form the fourth layer of enhanced features. Here, N represents the unified time step length of 16, and D represents the common feature dimension of the fusion module of 256.

7. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 1, characterized in that, The temporal dynamic weight generation module includes: a feature aggregation subunit, a temporal attention subunit, a weight generation subunit, and a weighted fusion subunit.

8. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 6, characterized in that, The temporal dynamic weight generation module processes the interactive features by: using a feature aggregation subunit to concatenate the four-layer features of each stream along the channel dimension to form a hierarchical feature sequence; using a temporal attention subunit to interact with the hierarchical feature sequence and evaluate the importance of the four-layer features at each time step; using a weight generation subunit to generate four-layer fusion weights for each time step based on the importance of the four-layer features, and normalizing them using Softmax to output a temporal dynamic weight matrix; and using a weighted fusion subunit to perform a weighted summation of the four-layer features based on the generated temporal dynamic weights to output the final enhanced features for each stream.

9. The four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 1, characterized in that, The temporal pooling module adopts a multi-scale temporal pooling structure, including a global attention pooling branch, a local convolutional pooling branch, and a temporal attention pooling branch; and performs adaptive weighted fusion on the outputs of each branch.

10. A four-stream human action recognition method based on multimodal hierarchical adaptive fusion according to claim 1, characterized in that, The model's loss function includes the main classification loss, the auxiliary classification loss, and the feature regularization term.

Images