A human posture estimation method and system based on Mamba dual-mode signal fusion

Abstract

The application discloses a WiFi-IMU dual-mode human body posture estimation method and system based on Mamba, belonging to the field of multi-modal perception and posture estimation. With CSI and IMU dual-mode signals as input, spatial features are extracted through grouped deep separable convolution, and different layers of bidirectional Mamba are used to adapt to sampling rate differences. Through content adaptive gating, dual-mode weighted fusion and noise suppression are realized. In combination with hierarchical biomechanical skeleton modeling, joint identity embedding and graph attention injection human body topology prior, fine residual correction of end joints is carried out. Then, heat map decoding and kinematic constraint fine tuning are carried out, and compliant two-dimensional joint coordinates are output. The application does not need visual collection and has strong privacy. The accuracy of end joints and physical rationality are significantly improved. In experiments, PCK@20 reaches 96.24%, MPJPE is 17.3 pixels, and Ext@20 is 91.40%, which is better than single-mode and traditional methods, and is suitable for indoor monitoring, medical rehabilitation, human-computer interaction and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal perception technology, specifically to a method and system for human pose estimation based on Mamba dual-modal signal fusion, which is particularly suitable for human pose tracking and behavior analysis in indoor environments where privacy needs to be protected. Background Technology

[0002] Human pose estimation (HPE) has broad application prospects in fields such as intelligent surveillance, medical rehabilitation, virtual reality, and human-computer interaction. Traditional vision-based HPE methods have achieved high accuracy in standard environments. However, vision-based methods have inherent limitations in privacy-sensitive locations and under non-ideal conditions such as low light or severe occlusion. Therefore, researchers have begun to explore privacy-preserving HPE methods based on non-visual sensors.

[0003] Among non-visual sensors, WiFi Channel State Information (CSI) and Inertial Measurement Units (IMUs) are two highly promising modalities. CSI, by analyzing the multipath effects of wireless signals reflected off the human body surface, enables contactless macroscopic motion perception, naturally protecting privacy and unaffected by lighting conditions. IMUs, through accelerometers and gyroscopes worn on the limbs, acquire high-precision local dynamic signals, making them particularly sensitive to rapid and subtle movements. Theoretically, their complementarity is very evident: CSI provides a coarse outline of the global skeleton, while IMU provides fine-grained changes in joint angles. Effective fusion of the two could potentially achieve near-visual-level accuracy while preserving privacy.

[0004] However, no mature solution has yet emerged in the current technology that can efficiently fuse CSI and IMU dual-modal signals for human pose estimation. The main reasons are as follows: First, the physical characteristics of the two signals are heterogeneous—CSI reflects the scattering and interference of electromagnetic waves, while IMU reflects acceleration and angular velocity in Newtonian mechanics. Direct splicing or simple weighted fusion introduces a large amount of noise. Second, the sampling rates of the two differ significantly: the CSI sampling rate of commercial WiFi network cards can typically reach 600Hz or even higher, while the sampling rate of common IMU modules is mostly around 100Hz, resulting in serious timing misalignments and discrepancies. Third, most existing methods ignore the inherent biomechanical topology of the human skeleton when decoding human pose from fused features, leading to non-physical artifacts in the output pose, such as inconsistent bone lengths and joint orientations that violate the laws of human movement, especially with larger errors at high-degree-of-freedom end joints such as the wrist and ankle. Fourth, there is still a lack of publicly available datasets and standardized evaluation methods for CSI and IMU dual-modal fusion, limiting research progress in this area.

[0005] Therefore, there is an urgent need to provide a pose estimation method that can adaptively align bimodal temporal sequences, explicitly inject prior knowledge of the human skeleton, and finely correct the end joints. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of the prior art and provide a method and system for human posture estimation based on Mamba dual-modal signal fusion. The method takes WiFi channel state information (CSI) signal and IMU inertial measurement signal as inputs, and outputs two-dimensional coordinates of 15 target joints of the human body through six execution steps: dual-modal coding, adaptive cross-modal fusion, hierarchical skeleton modeling, fine-grained correction of end joints, heat map decoding, and kinematic constraint correction. The two input signals are pre-aligned in time, and the CSI signal and IMU signal each contain multiple frames of sampled data and each has independent feature dimensions.

[0007] In a first aspect, the present invention provides a human pose estimation method based on Mamba dual-modal signal fusion, specifically comprising the following steps: Step S1: Independently encode the dual-modal signals to generate global features of a unified dimension.

[0008] Specifically, this step constructs a CSI encoder and an IMU encoder respectively, performs independent spatial feature extraction and timing modeling on the two heterogeneous time-series signals, and compresses signals with different dimensions and physical characteristics into a global feature vector with a unified feature dimension D.

[0009] First, grouped depthwise separable convolutions are used as the spatial feature extraction front-end for the two encoders. These depthwise separable convolutions include depthwise convolutions and pointwise convolutions; the formula for calculating the depthwise convolution is as follows: In the formula: For activation function, For learnable bias terms, No. The input of the group (sequence length is) ), The learnable convolutional kernel weights for this group, This is the output for this group.

[0010] The expression for pointwise convolution is as follows: in For the pointwise convolution kernel, the first The weight of each channel, For all splicing, output From spatial dimensions Transform to To adapt to the different transformation requirements of CSI data and IMU data.

[0011] Specifically, the CSI input signal contains 9 pairs of antennas, each pair with 30 subcarriers. After fusing amplitude and phase information, a 540-dimensional feature vector is formed per frame. Cross-link-subcarrier feature fusion is completed through two layers of depthwise separable convolution, achieving dimensionality compression and multipath response mode extraction. The IMU input signal contains 10 sensing units, each outputting 6-axis data. The feature dimension of a single frame is 60-dimensional. It is mapped to 128-dimensional and 256-dimensional dimensions through two layers of depthwise separable convolution, achieving alignment with the CSI feature dimension.

[0012] Secondly, after spatial feature extraction, bidirectional Mamba blocks are used to complete temporal dynamic modeling, and a selective state-space model is used to capture long-distance temporal dependencies of the signal; the unidirectional Mamba block calculation method is as follows: in This indicates frame-by-frame block input. As a potential state, These are the learnable residual weights. The input dependency matrix is ​​calculated as follows: in The SiLU activation function is used. This is the learnable projection parameter matrix.

[0013] The formulas for calculating the forward hidden state and the reverse hidden state of the bidirectional Mamba module, which traverses the sequence in both the forward and reverse directions, are as follows: The two hidden states are merged through element-wise addition: To obtain global sequence representation, the encoder adds learnable token vectors at the beginning and end of the sequence, aggregates bidirectional Mamba output features, and takes the average as the modality global feature; the CSI encoder stacks 4 bidirectional Mamba blocks and the IMU encoder stacks 2 bidirectional Mamba blocks to compensate for the sampling rate difference between the two types of sensors and achieve dynamic temporal alignment.

[0014] Step S2: Content-adaptive gating fusion to construct a dual-modal fusion feature vector.

[0015] This step is based on a gated fusion mechanism, dynamically generating fusion weights according to the feature content of the two modalities to achieve adaptive cross-modal fusion. First, two independent gated networks are used to map the global features of CSI and IMU to modality confidence scores, where m represents the modality; the confidence score calculation formula is as follows: The confidence scores were then normalized using the Softmax function to obtain the complementary fusion weights: The weighted modal features are concatenated and input into a two-layer fully connected network with layer normalization to generate the final fused vector. The calculation formula is as follows: This gating mechanism can dynamically adjust the modal contribution level based on the motion state and signal-to-noise ratio, and features lightweight design, prevention of attention collapse, and strong self-adaptability.

[0016] Step S3: Layered skeleton modeling, integrating prior knowledge of human anatomy to generate structured joint features.

[0017] This step maps the fused vectors into a joint feature matrix with anatomical semantics. Through hierarchical partitioning, joint embedding, hierarchical conditional refinement, and graph attention optimization, it injects human biomechanical topological constraints. Specifically, it includes the following sub-steps: S3.1, Hierarchical Perceptual Projection: The 15 joints of the human body are divided into four biomechanical levels: core layer, trunk layer, limb layer, and endpoint layer. Each level uses an independent transformation function to complete feature projection. in It consists of a linear layer, layer normalization, and GELU activation function, and extracts joint motion features at different levels in a differentiated manner.

[0018] S3.2 Joint Identity Embedding: Learnable joint embedding vectors are introduced to add unique feature biases to homogeneous joints within the same level. The calculation formula is as follows: For each joint, there is an embedding vector. All embeddings form a topological parameter matrix, enabling the parameterized injection of anatomical prior knowledge.

[0019] S3.3, Hierarchical Gating Fusion: Constructing bilinear gating units based on hierarchical features and global features. The concatenation of these features is used as input, followed by sigmoid activation, dynamically balancing hierarchical features and globally fused features, and a gating vector. Generate adaptive joint features using the following formula: S3.4 Hierarchical Conditional Refinement: Refinement is performed layer by layer according to the kinematic dependency order of core layer → trunk layer → limb layer → endpoint layer. Upstream joint features are aggregated to generate conditional vectors, and gating coefficients are used to constrain the update of downstream joint features. For non-core layers... By aggregating the precursor layer Calculate the global condition vector based on features The calculation formula is as follows: This vector aggregates the pose intentions of upstream joints, providing a macroscopic reference for fine-grained localization of downstream joints. It uses gating coefficients based on Sigmoid activation. Control the strength of the influence of this conditional constraint on the current layer: in This represents the average feature of the current layer. Finally, the conditional features are updated as follows: Progressive refinement generates a feature matrix that incorporates the skeleton topology. This improves spatial consistency and accuracy.

[0020] S3.5, Skeleton Graph Attention Optimization: A topology graph is constructed with 15 joints as nodes. A 2-hop connectivity mask is introduced to build an enhanced multi-head graph attention network. The skeleton association strength is encoded by learnable edge bias. The attention scores between joints are calculated and the feature weighting is completed. Three attention layers are stacked to capture both vertical hierarchical dependencies and horizontal skeleton constraints.

[0021] Specifically, its core is a learnable edge bias. A multi-layer, multi-head masked graph attention mechanism is used, where edge biases are used to encode the prior association strength of different skeleton connections. For the first... Layer The attention score between two different nodes is calculated as follows: in and The first Head joint and Query and key vector, For each head's feature dimension, It is a 2-hop mask (0 if j is within the 2-hop range of i, otherwise 0). ). No. Head joint The output is a weighted sum of the node value vectors: in express The set of joints within the 2-jump range (including itself). All The outputs of each unit are spliced ​​together and then linearly projected, followed by residual connections and a 4x extended MLP.

[0022] Step S4: End-joint residual correction to optimize the prediction accuracy of high-degree-of-freedom endpoints.

[0023] Specifically, for four types of high-degree-of-freedom end joints such as the wrist and ankle, this step constructs a residual correction module based on cross-attention and multilayer perceptron. Using the elbow and knee parent joints as kinematic anchors, the motion association between the end joints and proximal joints is first bound through four-head cross-attention. Then, a three-layer joint-specific MLP is used to generate residual correction values, which are combined with learnable gating weights to complete feature updates. The update formula is as follows: Among them, the gating weight Learnable parameters are used to control the update intensity. Learnable weights suppress unnecessary updates to already accurate joints while allowing for stronger corrections to faulty joints.

[0024] This step differentiates and corrects errors at different end joints, suppresses end joint drift, and ensures overall skeleton posture consistency.

[0025] Step S5: Decode the heat map to generate the original two-dimensional coordinates of the joints.

[0026] The optimized joint feature matrix is ​​input into the heatmap decoder, and spatial confidence heatmaps for each joint are obtained through a two-layer MLP mapping. Gaussian distribution is used to construct training supervision labels, and the heatmaps are decoded using a Soft-Argmax differentiable regression algorithm. After temperature-scaled Softmax normalization, each heatmap is further normalized by temperature scaling to obtain a probability map. Final subpixel coordinates The spatial expectation is calculated using the following formula: Step S6: Fine-tune kinematic constraints to output a compliant human posture skeleton.

[0027] A kinematic constraint correction layer is constructed to learn the standard bone length and deviation threshold for each bone edge of the skeleton and calculate the predicted bone length error. Dynamic correction weights are generated based on the peak confidence of the joint heatmap. Coordinate fine-tuning is generated through a two-layer MLP, limiting the correction amplitude to no more than 0.1. Fine-tuning is only performed on joints with low confidence and abnormal errors. Finally, high-precision two-dimensional coordinates of 15 human joints are output to form a complete human posture skeleton without distortion artifacts.

[0028] A second aspect of the present invention provides a human pose estimation system based on Mamba dual-modal signal fusion, used to implement the pose estimation method described in any one of the above claims. The system includes: a signal acquisition module, a data preprocessing module, a dual-modal encoding module, an adaptive fusion module, a hierarchical skeleton modeling module, an end-joint refinement module, a heatmap decoding module, a kinematic constraint correction module, and a pose output module.

[0029] 1. Signal Acquisition Module: Used to acquire WiFi channel status information (CSI) signals and wearable IMU inertial measurement signals in indoor environments; 2. Data preprocessing module: Performs denoising, time alignment, and dimension normalization on two heterogeneous signals to generate standardized input samples; 3. Dual-modal coding module: Includes CSI encoder and IMU encoder, which rely on grouped depthwise separable convolution and bidirectional Mamba blocks to complete spatial feature extraction and temporal modeling, and output global features of two modalities; 4. Adaptive Fusion Module: Built-in content adaptive gating fusion unit, dynamically calculates modal fusion weights, and generates dual-modal fusion features; 5. Hierarchical skeleton modeling module: used to complete joint hierarchy division, identity embedding, hierarchy condition refinement and graph attention feature optimization, and inject human anatomical priors; 6. End-joint refinement module: Through the cross-attention residual correction unit, secondary error correction is performed on the wrist and ankle end-joints; 7. Heatmap Decoding Module: Generates joint confidence heatmaps and uses the Soft-Argmax algorithm to regress and obtain the original two-dimensional coordinates of the joints; 8. Kinematic Constraint Correction Module: Fine-tunes joint coordinates based on bone length constraints to eliminate non-physical posture artifacts; 9. Posture Output Module: Visualizes and outputs the posture of a two-dimensional human skeleton containing 15 human joints.

[0030] Compared with the prior art, the present invention has the following beneficial effects: First, this invention uses a bidirectional Mamba state-space model to complete the time-series modeling of heterogeneous signals, adapting to the differences in CSI and IMU sampling rates. Compared with Transformer and recurrent neural networks, it has a lower parameter count and longer time-series modeling capability, effectively extracting weak human motion features.

[0031] Second, a lightweight content adaptive gating fusion mechanism is designed to dynamically adjust the dual-modal weights based on the signal-to-noise ratio and motion state, thereby solving the problems of high noise and modal redundancy interference in heterogeneous signal fusion.

[0032] Third, it pioneered a layered skeleton modeling architecture, combining biomechanical hierarchical division, joint embedding and graph attention network to explicitly inject human anatomical priors, thereby suppressing non-physical artifacts such as skeletal deformities and joint rotations from the root.

[0033] Fourth, a dedicated correction module for end joints is added, using proximal joints as anchor points to constrain high-degree-of-freedom end-point movements, significantly reducing prediction errors for wrists and ankles.

[0034] Fifth, a post-kinematic constraint correction layer is applied, which fine-tunes the coordinates with the physiological length of the skeleton as the boundary, improving the physical rationality of the posture while ensuring the accuracy of the model. This invention adopts a non-visual perception method throughout the process to protect user privacy, and is suitable for privacy-sensitive scenarios such as home monitoring, medical rehabilitation, and human-computer interaction in closed indoor industrial settings. Attached Figure Description

[0035] Figure 1 This is a flowchart of a method and system for human pose estimation based on Mamba dual-modal signal fusion according to the present invention. Figure 2 This is an overview diagram of the HSMamba-HPE architecture of the present invention; Figure 3 This is a schematic diagram of the joint estimation of human posture based on WiFi-CSI heatmap and IMU waveform according to the present invention; Figure 4 This is a functional block diagram of the system of the present invention.

[0036] Note: Attached Figure 3 The rows from top to bottom are: CSI amplitude heatmap, corresponding six-axis IMU waveform (10 nodes), the real skeleton obtained by OpenPose, and the predicted pose. The eight action sequences from left to right are: walking, chest expansion, left leg kick, waving, right leg kick, arm raising, walking, and turning. Detailed Implementation

[0037] The following is in conjunction with the appendix Figure 1 To be continued Figure 4 The technical solution of the present invention will be further described in detail below with specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiment.

[0038] Example 1 See attached document Figure 1 A method for human pose estimation based on Mamba dual-modal signal fusion includes the following steps: Step S1: Independently encode the dual-modal signals to generate global features of a unified dimension.

[0039] Specifically, this step constructs a CSI encoder and an IMU encoder respectively, performs independent spatial feature extraction and timing modeling on the two heterogeneous time-series signals, and compresses signals with different dimensions and physical characteristics into a global feature vector with a unified feature dimension D.

[0040] First, grouped depthwise separable convolutions are used as the spatial feature extraction front-end for the two encoders. These depthwise separable convolutions include depthwise convolutions and pointwise convolutions; the formula for calculating the depthwise convolution is as follows: In the formula: For activation function, For learnable bias terms, No. The input of the group (sequence length is) ), For the learnable convolutional kernel weights of this group, This is the output for this group.

[0041] The expression for pointwise convolution is as follows: in For the pointwise convolution kernel, the first The weight of each channel, For all splicing, output From spatial dimensions Transform to To adapt to the different transformation requirements of CSI data and IMU data.

[0042] Specifically, the CSI input signal contains 9 pairs of antennas, each pair with 30 subcarriers. After fusing amplitude and phase information, a 540-dimensional feature vector is formed per frame. Cross-link-subcarrier feature fusion is completed through two layers of depthwise separable convolution, achieving dimensionality compression and multipath response mode extraction. The IMU input signal contains 10 sensing units, each outputting 6-axis data. The feature dimension of a single frame is 60-dimensional. It is mapped to 128-dimensional and 256-dimensional dimensions through two layers of depthwise separable convolution, achieving alignment with the CSI feature dimension.

[0043] Secondly, after spatial feature extraction, bidirectional Mamba blocks are used to complete temporal dynamic modeling, and a selective state-space model is used to capture long-distance temporal dependencies of the signal; the unidirectional Mamba block calculation method is as follows: in This indicates frame-by-frame block input. As a potential state, These are the learnable residual weights. The input dependency matrix is ​​calculated as follows: in The SiLU activation function is used. This is the learnable projection parameter matrix.

[0044] The formulas for calculating the forward hidden state and the reverse hidden state of the bidirectional Mamba module, which traverses the sequence in both the forward and reverse directions, are as follows: The two hidden states are merged through element-wise addition: To obtain global sequence representation, the encoder adds learnable token vectors at the beginning and end of the sequence, aggregates bidirectional Mamba output features, and takes the average as the modality global feature; the CSI encoder stacks 4 bidirectional Mamba blocks and the IMU encoder stacks 2 bidirectional Mamba blocks to compensate for the sampling rate difference between the two types of sensors and achieve dynamic temporal alignment.

[0045] Step S2: Content-adaptive gating fusion to construct a dual-modal fusion feature vector.

[0046] This step is based on a gated fusion mechanism, dynamically generating fusion weights according to the feature content of the two modalities to achieve adaptive cross-modal fusion. First, two independent gated networks are used to map the global features of CSI and IMU to modality confidence scores, where m represents the modality; the confidence score calculation formula is as follows: The confidence scores were then normalized using the Softmax function to obtain the complementary fusion weights: The weighted modal features are concatenated and input into a two-layer fully connected network with layer normalization to generate the final fused vector. The calculation formula is as follows: This gating mechanism can dynamically adjust the modal contribution level based on the motion state and signal-to-noise ratio, and features lightweight design, prevention of attention collapse, and strong self-adaptability.

[0047] Step S3: Layered skeleton modeling, integrating prior knowledge of human anatomy to generate structured joint features.

[0048] This step maps the fused vectors into a joint feature matrix with anatomical semantics. Through hierarchical partitioning, joint embedding, hierarchical conditional refinement, and graph attention optimization, it injects human biomechanical topological constraints. Specifically, it includes the following sub-steps: S3.1, Hierarchical Perceptual Projection: The 15 joints of the human body are divided into four biomechanical levels: core layer, trunk layer, limb layer, and endpoint layer. Each level uses an independent transformation function to complete feature projection. in It consists of a linear layer, layer normalization, and GELU activation function, and extracts joint motion features at different levels in a differentiated manner.

[0049] S3.2 Joint Identity Embedding: Learnable joint embedding vectors are introduced to add unique feature biases to homogeneous joints within the same level. The calculation formula is as follows: For each joint, there is an embedding vector. All embeddings form a topological parameter matrix, enabling the parameterized injection of anatomical prior knowledge.

[0050] S3.3, Hierarchical Gating Fusion: Constructing bilinear gating units based on hierarchical features and global features. The concatenation of these features is used as input, followed by sigmoid activation, dynamically balancing hierarchical features and globally fused features, and a gating vector. Generate adaptive joint features using the following formula: S3.4 Hierarchical Conditional Refinement: Refinement is performed layer by layer according to the kinematic dependency order of core layer → trunk layer → limb layer → endpoint layer. Upstream joint features are aggregated to generate conditional vectors, and gating coefficients are used to constrain the update of downstream joint features. For non-core layers... By aggregating the precursor layer Calculate the global condition vector based on features The calculation formula is as follows: This vector aggregates the pose intentions of upstream joints, providing a macroscopic reference for fine-grained localization of downstream joints. It uses gating coefficients based on Sigmoid activation. Control the strength of the influence of this conditional constraint on the current layer: in This represents the average feature of the current layer. Finally, the conditional features are updated as follows: Progressive refinement generates a feature matrix that incorporates the skeleton topology. This improves spatial consistency and accuracy.

[0051] S3.5, Skeleton Graph Attention Optimization: A topology graph is constructed with 15 joints as nodes. A 2-hop connectivity mask is introduced to build an enhanced multi-head graph attention network. The skeleton association strength is encoded by learnable edge bias. The attention scores between joints are calculated and the feature weighting is completed. Three attention layers are stacked to capture both vertical hierarchical dependencies and horizontal skeleton constraints.

[0052] Specifically, its core is a learnable edge bias. A multi-layer, multi-head masked graph attention mechanism is used, where edge biases are used to encode the prior association strength of different skeleton connections. For the first... Layer The attention score between two different nodes is calculated as follows: in and The first Head joint and Query and key vector, For each head's feature dimension, It is a 2-hop mask (0 if j is within the 2-hop range of i, otherwise 0). ). No. Head joint The output is a weighted sum of the node value vectors: in express The set of joints within the 2-jump range (including itself). All The outputs of each unit are spliced ​​together and then linearly projected, followed by residual connections and a 4x extended MLP.

[0053] Step S4: End-joint residual correction to optimize the prediction accuracy of high-degree-of-freedom endpoints.

[0054] Specifically, for four types of high-degree-of-freedom end joints such as the wrist and ankle, this step constructs a residual correction module based on cross-attention and multilayer perceptron. Using the elbow and knee parent joints as kinematic anchors, the motion association between the end joints and proximal joints is first bound through four-head cross-attention. Then, a three-layer joint-specific MLP is used to generate residual correction values, which are combined with learnable gating weights to complete feature updates. The update formula is as follows: Among them, the gating weight Learnable parameters are used to control the update intensity. Learnable weights suppress unnecessary updates to already accurate joints while allowing for stronger corrections to faulty joints.

[0055] This step differentiates and corrects errors at different end joints, suppresses end joint drift, and ensures overall skeleton posture consistency.

[0056] Step S5: Decode the heat map to generate the original two-dimensional coordinates of the joints.

[0057] The optimized joint feature matrix is ​​input into the heatmap decoder, and spatial confidence heatmaps for each joint are obtained through a two-layer MLP mapping. Gaussian distribution is used to construct training supervision labels, and the heatmaps are decoded using a Soft-Argmax differentiable regression algorithm. After temperature-scaled Softmax normalization, each heatmap is further normalized by temperature scaling to obtain a probability map. Final subpixel coordinates The spatial expectation is calculated using the following formula: Step S6: Fine-tune kinematic constraints to output a compliant human posture skeleton.

[0058] A kinematic constraint correction layer is constructed to learn the standard bone length and deviation threshold for each bone edge of the skeleton and calculate the predicted bone length error. Dynamic correction weights are generated based on the peak confidence of the joint heatmap. Coordinate fine-tuning is generated through a two-layer MLP, limiting the correction amplitude to no more than 0.1. Fine-tuning is only performed on joints with low confidence and abnormal errors. Finally, high-precision two-dimensional coordinates of 15 human joints are output to form a complete human posture skeleton without distortion artifacts.

[0059] See attached document Figure 4The second aspect of the present invention provides a human pose estimation system based on Mamba dual-modal signal fusion, used to implement the pose estimation method described in any one of the above claims. The system includes: a signal acquisition module, a data preprocessing module, a dual-modal encoding module, an adaptive fusion module, a hierarchical skeleton modeling module, an end-joint refinement module, a heatmap decoding module, a kinematic constraint correction module, and a pose output module.

[0060] 1. Signal Acquisition Module: Used to acquire WiFi channel status information (CSI) signals and wearable IMU inertial measurement signals in indoor environments; 2. Data preprocessing module: Performs denoising, time alignment, and dimension normalization on two heterogeneous signals to generate standardized input samples; 3. Dual-modal coding module: Includes CSI encoder and IMU encoder, which rely on grouped depthwise separable convolution and bidirectional Mamba blocks to complete spatial feature extraction and temporal modeling, and output global features of two modalities; 4. Adaptive Fusion Module: Built-in content adaptive gating fusion unit, dynamically calculates modal fusion weights, and generates dual-modal fusion features; 5. Hierarchical skeleton modeling module: used to complete joint hierarchy division, identity embedding, hierarchy condition refinement and graph attention feature optimization, and inject human anatomical priors; 6. End-joint refinement module: Through the cross-attention residual correction unit, secondary error correction is performed on the wrist and ankle end-joints; 7. Heatmap Decoding Module: Generates joint confidence heatmaps and uses the Soft-Argmax algorithm to regress and obtain the original two-dimensional coordinates of the joints; 8. Kinematic Constraint Correction Module: Fine-tunes joint coordinates based on bone length constraints to eliminate non-physical posture artifacts; 9. Posture Output Module: Visualizes and outputs the posture of a two-dimensional human skeleton containing 15 human joints.

[0061] Example 2 This embodiment was trained and evaluated on a self-collected synchronous bimodal dataset. (See attached document.) Figure 2 With appendix Figure 3As shown, the dataset contains WiFi CSI and IMU data, collected from seven subjects during eight daily activities. CSI data was acquired using an Intel 5300 network card with nine pairs of antennas at a sampling rate of 600 Hz. IMU data was simultaneously acquired from ten six-axis sensors worn by the subjects at a sampling rate of 100 Hz, eight of which were mounted on major joints of the limbs, one on the torso, and one on the head. Human motion posture target values ​​(15 2D key points) were obtained using OpenPose with a time-stamped synchronized camera. After time windowing, the self-built dataset contains approximately 376,000 synchronized CSI-IMU-pose label samples.

[0062] Model performance was evaluated using the percentage of correct keypoints (PCK) and the mean joint position error (MPJPE). This embodiment reports PCK@α, which is the proportion of predicted joint errors less than α times the length of the torso segment, where α takes values ​​of 0.1, 0.2, and 0.5. MPJPE is the average Euclidean distance (in pixels) between all predicted joint coordinates and the true joint coordinates. Additionally, Ext@20 was used to measure the average accuracy of end joints (wrist and ankle) with a PCK threshold of 0.2.

[0063] The proposed model is implemented using PyTorch. The model training uses the Adam optimizer, with weight decay and an initial learning rate of 0.001. The batch size is 256, the training is conducted for 100 epochs, and the input time window is divided into granularities of 0.2 seconds. The model is trained on a workstation equipped with a single Nvidia RTX 6000ada (48G) processor.

[0064] In the performance evaluation comparative experiments, the complete HSMamba-HPE model was compared with models using only CSI data, WiSPPN, IMU data only, and Transformer-based models. The overall algorithm performance evaluation results are shown in Table 1. The results show that HSMamba-HPE significantly outperforms the selected reference model. Compared with single-modal data, WiFi data and IMU data have significant complementarity in human pose recognition tasks. The multimodal fusion strategy can effectively integrate the advantages of two data sources, achieving significant performance improvements compared to any single modality. Specifically, the MPJPE of the complete model is 17.3 pixels, PCK@20 is 96.24%, and Ext@20 is 91.40%.

[0065] Table 1 Overall Performance Comparison To better illustrate the specific differences between single-modal and multimodal data, the recognition performance of each key point at PCK@20 is presented separately. The results are shown in Table 2. The vast majority of key points indicate that multimodal data has a higher recognition accuracy than single-modal data.

[0066] Table 2 Results of each key point The cross-user comparison experiment used cross-validation, selecting six subjects in rotation as the test set. The results are shown in Table 3. The experimental results indicate that different subjects showed varying strengths and weaknesses across different metrics, demonstrating significant differences among subjects at the dataset level, which aligns with the criteria for cross-user datasets.

[0067] Table 3 Comparison of Fusion Mechanisms In the ablation study during the hierarchical skeleton modeling stage, the contributions of three key components were ablated: hierarchical projection, joint embedding, and adaptive gating. Five progressive strategies were used: a baseline model (replicating global features to all joints), hierarchical projection only (setting a dedicated projection layer for each anatomical level, mapping global features to corresponding joints according to hierarchical relationships, without embedding or gating), joint embedding only (using shared projection plus learnable joint position embedding, without gating), gating fusion only (adaptive gating between global features and uniform joint features), and the complete model (combining hierarchical projection, joint embedding, and gating fusion). The results are shown in Table 4. The baseline model performed poorly, confirming the inadequacy of the naive feature replication method. Adding hierarchical projection brought a significant improvement, indicating that hierarchical-specific mapping effectively guides the information flow. Adding joint embedding produced the best single-module results, validating the importance of learnable joint embedding for discriminative representation. Using gating alone provided only a small improvement, possibly because adaptive weighting depends on meaningful joint-specific features. The complete model achieved the highest accuracy, demonstrating that the three components are complementary, and the absence of any one of them leads to performance degradation.

[0068] Table 4 Evaluation Indicators for Different Ablation Protocols In the comparison of fusion mechanisms, two benchmark fusion strategies are compared with the proposed content-adaptive gating mechanism: simple concatenation of CSI and IMU features, and alignment of temporal dimensions using 1D convolution. Performance comparison results are shown in Table 5. The gating fusion mechanism achieves the best performance, outperforming simple concatenation by approximately 13.3% and 1D convolution by approximately 2.6% in PCK@20, demonstrating its effectiveness in handling dynamic modality reliability and temporal misalignment.

[0069] Table 5 Comparison of Fusion Mechanisms The proposed HSMamba-HPE model transforms globally uniform features into anatomically aware structured joint representations through a bimodal spatiotemporal feature extraction mechanism based on bidirectional Mamba and a hierarchical skeleton modeling module. The model integrates a multi-head graph attention network to capture lateral skeleton constraints and introduces an endpoint refinement module for adaptive correction of the wrist and ankle joints. End-to-end training under visual supervision achieves high-precision and physically plausible pose estimation, outperforming existing benchmark algorithms.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for human pose estimation based on Mamba dual-modal signal fusion, characterized in that: Includes the following steps: Step S1: Perform dual-modal independent encoding on the input WiFiCSI signal and IMU signal, extract spatial features through grouped depthwise separable convolution, and use bidirectional Mamba blocks for temporal modeling to generate dual-modal global features of a unified dimension. Step S2: Perform content-adaptive gating fusion on the dual-modal global features, dynamically generate fusion weights based on the modal feature content, and obtain the dual-modal fusion feature vector; Step S3: Based on the human biomechanical hierarchy, perform hierarchical skeleton modeling of the fused features. Through hierarchical perceptual projection, joint identity embedding and adaptive gating fusion, generate structured joint features with anatomical priors, and use graph attention network to constrain the skeleton topology. Step S4: Perform residual correction on the distal joints of the wrist and ankle. Using the parent joint as the anchor point, generate correction values ​​through cross-attention and MLP to optimize the prediction accuracy of the distal joints. Step S5: Input the refined joint features into the heatmap decoder and obtain the initial two-dimensional joint coordinates through SoftArgmax regression; Step S6: Fine-tune the initial coordinates based on human kinematic constraints, limit the correction range, and output a human posture skeleton that conforms to physical rationality.

2. The method for human pose estimation based on Mamba dual-modal signal fusion according to claim 1, characterized in that, In step S1, the CSI encoder stacks 4 bidirectional Mamba blocks and the IMU encoder stacks 2 bidirectional Mamba blocks to adapt to the sampling rate difference between 600Hz and 100Hz, and learnable tokens are added at the beginning and end of the sequence to aggregate global temporal features.

3. The method for human pose estimation based on Mamba dual-modal signal fusion according to claim 1, characterized in that, Step S2, the content adaptive gating fusion, includes: generating CSI and IMU confidence scores through an independent gating network, normalizing them to complementary weights using Softmax; concatenating the weighted features and outputting a fusion vector through two fully connected layers and layer normalization, thereby achieving adaptive fusion that emphasizes CSI for high signal-to-noise ratio and IMU for fast motion.

4. The human pose estimation method based on Mamba dual-modal signal fusion according to claim 1, characterized in that, In step S3, the 15 joints of the human body are divided into four biomechanical levels: core layer, trunk layer, limb layer, and endpoint layer. The hierarchical features are obtained through hierarchical perception projection, and learnable joint identity embedding is added to distinguish joints in the same layer. The hierarchical features and global features are dynamically balanced through gating.

5. The method for human pose estimation based on Mamba dual-modal signal fusion according to claim 1, characterized in that, Step S3 employs a multi-head graph attention network with a 2-hop connectivity mask, using joints as nodes and bones as edges, and introduces learnable edge biases to model joint associations, forcing poses to conform to human kinematic constraints.

6. The method for human pose estimation based on Mamba dual-modal signal fusion according to claim 1, characterized in that, In step S4, the end-joint correction uses a 4-head cross-attention binding to the parent joint context and generates residuals through a joint-specific 3-layer MLP, with the correction intensity controlled by learnable gating weights.

7. The method for human pose estimation based on Mamba dual-modal signal fusion according to claim 1, characterized in that, Step S6 learns the standard bone length calculation error, assigns correction weights based on the confidence level of the heatmap, and limits the coordinate fine-tuning range to within 0.1, correcting only low-confidence joints.

8. A Mamba-based dual-modal signal fusion human pose estimation system, used to implement the method of any one of claims 17, characterized in that, include: The signal acquisition module is used to simultaneously acquire WiFi CSI signals and IMU inertial signals; The data preprocessing module is used for signal denoising, time alignment, and dimension normalization. A bimodal coding module is used to extract bimodal spatiotemporal features through depthwise separable convolution and bidirectional Mamba; The adaptive fusion module is used to perform content adaptive gating fusion to obtain fused features; A hierarchical skeleton modeling module is used for hierarchical projection, joint embedding, and graph attention optimization. The end-joint refinement module is used to perform residual correction on the wrist and ankle endpoints; The heatmap decoding module is used to generate joint heatmaps and regress two-dimensional coordinates; The kinematic constraint correction module is used to fine-tune the posture based on bone length constraints; The posture output module is used to output a 2D human skeleton with 15 joints.

Images