Human pose estimation method and device based on visual-inertial fusion

Abstract

The application relates to the technical field of three-dimensional human posture estimation, in particular to a human posture estimation method and device based on visual-inertial fusion. The method is characterized in that a network for feature extraction is designed for visual data and inertial measurement data respectively, the original data is converted to obtain first visual features and first inertial features, and then the first visual features and the first inertial features are fused to obtain first fusion features. The use of the fourth network makes the output second fusion features more smooth and reliable. The global fusion features obtained by the second fusion features according to multiple groups contain all the information of the human posture. The method using the self-attention space-time mamba network structure and the network structure with multi-head self-attention or cross mamba can realize efficient single-modal feature extraction through selective state space modeling, complete deep fusion of heterogeneous modalities by using the cross-attention mechanism, and solve the problem that the current human posture estimation cannot balance the accuracy and efficiency.

Description

Technical Field

[0001] This application relates to the field of three-dimensional human pose estimation technology, and in particular to a human pose estimation method and device based on visual-inertial fusion. Background Technology

[0002] Multimodal 3D human pose estimation is a technique that simultaneously utilizes visual and inertial measurement unit (IMU) data to accurately reconstruct the position and motion trajectory of human joints in 3D space. It is widely used in motion capture, human-computer interaction, medical rehabilitation, and autonomous driving. Because visual modality is susceptible to occlusion, lighting variations, and scale blur, while inertial modality suffers from cumulative drift and initialization errors, the research and development focus of multimodal 3D human pose estimation lies in how to perform spatiotemporal alignment, feature extraction, and information fusion on these two heterogeneous, asynchronous signals with significantly different sampling rates.

[0003] Attention mechanisms and state-space models such as Transformer and Mamba are commonly used tools for cross-modal applications. However, the former's quadratic computational complexity limits its real-time application in resource-constrained scenarios, while the latter lags behind attention mechanisms in representing complex spatial relationships. Therefore, there is an urgent need to provide a multimodal 3D human pose estimation method that balances accuracy and efficiency. Summary of the Invention

[0004] In view of this, the present application provides a human pose estimation method and apparatus based on visual-inertial fusion, which solves the problem that current human pose estimation cannot balance accuracy and efficiency.

[0005] According to a first aspect of this application, a human pose estimation method based on visual-inertial fusion is provided, the method comprising:

[0006] The first network is used to process visual data to obtain first visual features;

[0007] The inertial measurement data is processed using a second network to obtain the first inertial feature;

[0008] The first fused feature is obtained by fusing the first visual feature and the first inertial feature based on the third network;

[0009] The first fusion feature is processed using a fourth network to obtain the second fusion feature;

[0010] Based on the second fusion features corresponding to multiple groups, global fusion features are obtained;

[0011] The global fusion features are expanded to obtain the first expanded features;

[0012] The fifth network, based on stacked deep layers, constructs a global model from the first expanded features to obtain the global model.

[0013] Human pose estimation is obtained based on a global model;

[0014] The first, second, fourth, and fifth networks have a self-attention spatiotemporal Mamba network structure;

[0015] The third network has a network structure with multi-head self-attention or cross-mamba.

[0016] According to a second aspect of this application, a human pose estimation device based on visual-inertial fusion is provided, the device comprising:

[0017] A visual data processing module is used to process visual data using a first network to obtain first visual features;

[0018] An inertial data processing module is used to process inertial measurement data using a second network to obtain a first inertial feature;

[0019] The first fusion processing module is used to obtain the first fused feature by fusing the first visual feature and the first inertial feature based on the third network;

[0020] The second fusion processing module is used to process the first fusion feature using the fourth network to obtain the second fusion feature;

[0021] The global fusion processing module is used to obtain global fusion features based on the second fusion features corresponding to multiple groups;

[0022] The global feature extension module is used to extend the global fused features to obtain the first expanded features;

[0023] The global model building module is used to build a global model from the first expanded features based on the fifth network with stacked deep layers, thereby obtaining the global model.

[0024] The human pose estimation module is used to obtain an estimate of human pose based on a global model.

[0025] The first, second, fourth, and fifth networks have a self-attention spatiotemporal Mamba network structure;

[0026] The third network has a network structure with multi-head self-attention or cross-mamba.

[0027] A third aspect of this application discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0028] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0029] In this embodiment, feature extraction networks are designed for visual data and inertial measurement data respectively. The original data is transformed into first visual features and first inertial features. Then, based on a third network, the first visual features and the first inertial features are fused to obtain a first fused feature. This feature extraction method using a dedicated network can remove redundant and interfering information unrelated to human pose estimation from the heterogeneous original data. On the other hand, it can reduce the data dimensionality, reduce the computational overhead of fusion, and avoid noise interference in the fusion process. At the same time, the use of a fourth network makes the output second fused feature smoother and more reliable. Thus, the global fused feature obtained by grouping the second fused features into multiple groups contains all the information of human pose. A global model is constructed based on the expanded first features, which can represent human pose. This method, which uses a spatiotemporal Mamba network structure with self-attention and a network structure with multi-head self-attention or cross-Mamba, achieves efficient single-modal feature extraction through selective state space modeling and completes deep fusion of heterogeneous modalities using a cross-attention mechanism. This solves the problem of current human pose estimation that cannot balance accuracy and efficiency. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A flowchart illustrating a human pose estimation method based on visual-inertial fusion provided in one embodiment of this application;

[0032] Figure 2 A flowchart illustrating the PoseVIMamba algorithm model used in a human pose estimation method based on visual-inertial fusion provided in one embodiment of this application.

[0033] Figure 3 A schematic diagram showing the grouping of the human skeleton, 2D key points, and IMU in a human pose estimation method based on visual-inertial fusion provided in one embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the architecture of a feature extraction model in an embodiment of the human pose estimation method based on visual-inertial fusion provided in one embodiment of this application.

[0035] Figure 5This is a flowchart illustrating the module structure used in the feature fusion stage of the algorithm framework in the human pose estimation method based on visual-inertial fusion provided in one embodiment of this application.

[0036] Figure 6 This is a schematic diagram of the architecture of another embodiment of the feature extraction model in the human pose estimation method based on visual-inertial fusion provided in one embodiment of this application;

[0037] Figure 7 This is a block diagram of a human pose estimation device based on visual-inertial fusion, provided for another embodiment of this application. Detailed Implementation

[0038] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and are not intended to imply any limitation on the scope of the disclosure.

[0039] According to one embodiment of this application, a human pose estimation method based on visual-inertial fusion is provided, such as... Figure 1 As shown, the method includes: Step S101: Processing visual data using a first network to obtain a first visual feature; Step S102: Processing inertial measurement data using a second network to obtain a first inertial feature; Step S103: Fusing the first visual feature and the first inertial feature based on a third network to obtain a first fused feature; Step S104: Processing the first fused feature using a fourth network to obtain a second fused feature; Step S105: Obtaining a global fused feature based on the second fused features corresponding to multiple groups; Step S106: Expanding the global fused feature to obtain a first expanded feature; Step S107: Constructing a global model for the first expanded feature based on a fifth network with stacked deep layers to obtain a global model; Step S108: Obtaining an estimate of human pose based on the global model; The first network, the second network, the fourth network, and the fifth network have a self-attention spatiotemporal Mamba network structure; The third network has a multi-head self-attention or cross-Mamba network structure.

[0040] Specifically, data can be acquired from the target using a pre-set visual acquisition device and an inertial measurement device to obtain visual data and inertial measurement data. The inertial measurement device may include an IMU sensor mounted on the camera and / or an IMU sensor mounted on the target object. The visual data may be the result of extracting data from the original image acquired by the camera using a pre-built 2D keypoint extraction model.

[0041] Human movement is localized; the movement of the left arm is usually independent of the movement of the right leg. Grouping the human body's joints can significantly reduce model complexity and computational cost, making it easier for the model to learn effective local movement patterns. Specifically, the human body can be divided into five groups: left arm, right arm, left leg, right leg, and torso. Therefore, the fused features obtained through the fourth network are also grouped. In practical applications, the left and right arms can be further grouped according to the upper arm and forearm, and the left and right legs can be further grouped according to the thigh and calf.

[0042] Specifically, a global fusion feature can be obtained by concatenating the second fusion features of different groups, or it can be combined into a global feature vector using a preset pooling layer. The pooling layer can be a global max pooling layer or a global average pooling layer. For example, by averaging the second fusion features corresponding to each group, converting them to scalar values, and then concatenating them, the resulting global fusion feature will possess the most salient states of each key part (group) of the body.

[0043] Specifically, deconvolutional layers or interpolation upsampling layers can be used to spatially transform the global fusion features, thereby obtaining the coordinates of various key parts of the human body (such as groups).

[0044] This application implements feature extraction networks for visual data and inertial measurement data respectively, transforming the original data into first visual features and first inertial features. Then, based on a third network, the first visual features and the first inertial features are fused to obtain a first fused feature. This feature extraction method using a dedicated network can remove redundant and interfering information unrelated to human pose estimation from the heterogeneous original data, and reduce the data dimensionality to reduce the computational overhead of fusion, avoiding noise interference in the fusion process. At the same time, the use of a fourth network makes the output second fused feature smoother and more reliable. Thus, the global fused feature obtained by grouping the second fused features into multiple groups contains all the information of human pose. Based on the expanded first features, a global model is constructed, which can represent human pose, thereby improving the accuracy and robustness of human pose estimation and making up for the shortcomings of visual sensors being susceptible to occlusion and the accumulation of errors in inertial data over time.

[0045] In some embodiments, the first visual feature is obtained as follows: spatial location feature injection is applied to the visual keypoint features contained in the visual data, followed by layer normalization to obtain the first input feature; the first input feature is linearly projected onto the state space model to obtain the first tensor and the second tensor; the first tensor is convolved and gated to obtain the second feature; the second feature is expanded based on the rescanning and sorting of the second feature and the parent joint feature to obtain the third tensor; the third tensor is used as the input of the state space model, and a fourth tensor is obtained based on the second gating function and the second tensor; a fifth tensor is obtained based on the linear transformation of the first input feature and the fourth tensor; layer linear normalization is applied to the fifth tensor, and the normalized tensor is used as the input of the fully connected network to obtain the sixth tensor; the first visual feature is obtained based on the fifth tensor and the sixth tensor.

[0046] Specifically, after acquiring the image captured by the camera, a pre-set 2D keypoint detector can be used to process it to obtain a 2D keypoint sequence (i.e., visual data). Spatial location feature injection and layer normalization are then used to preprocess the 2D keypoints to determine the absolute position of each keypoint.

[0047] Specifically, the state-space model can be a structured state-space sequence model (S4), a state-space model (SSM), Mamba, etc. In this embodiment, the Mamba state-space model is used, and the first input feature is used as the input of the Mamba state-space model to obtain a tensor for calculating the state and a tensor for controlling the gating or output.

[0048] In some embodiments, the first inertial feature is obtained as follows: inertial measurement units are grouped to have an embedding dimension consistent with the visual branch; spatial location feature injection is applied to the inertial measurement data contained in the first inertial feature, followed by layer normalization to obtain a second input feature; the second input feature is linearly projected onto the state space model to obtain an eleventh tensor and a twelfth tensor; a thirteenth gating weight is generated based on the twelfth tensor; a thirteenth feature is obtained based on the convolution result of the thirteenth gating weight and the eleventh tensor in a preset direction; the thirteenth tensor is used as the input to the state space model, and a fourteenth tensor is obtained based on a second gating function; a fifteenth tensor is obtained based on the linear transformation of the eleventh input feature and the fourteenth tensor; layer linear normalization is applied to the fifteenth tensor, and the normalized tensor is used as the input to a fully connected network to obtain a sixteenth tensor; the first inertial feature is obtained based on the fifteenth tensor and the sixteenth tensor.

[0049] Specifically, 3D human pose estimation systems typically employ multiple IMU sensors. For example, assuming the system provided in this embodiment includes multiple Vicon cameras and 13 IMUs, these 13 IMUs are deployed at key locations such as the upper limbs, lower limbs, head, sternum, and pelvis. Therefore, the IMUs are grouped according to their key deployment locations, such as the upper limb group, lower limb group, etc. The visual branch processes 2D keypoints. Assuming there are 17 keypoints, each point is represented by a D_v-dimensional feature vector, so the dimension of the entire visual feature might be 17*D_v. The inertial branch, having multiple IMUs, each outputs multi-channel data (e.g., 6 channels for acceleration and angular velocity). To match the dimension with the visual features during subsequent fusion, the data from each IMU sensor needs to be mapped to a D_v-dimensional feature vector through a linear projection layer, making the number of sensors * D_v = 17*D_v. This dimensional alignment allows the subsequent third network to process features from both models in a symmetrical and fair manner, simplifying the fusion process and ensuring that the visual and inertial features are at similar numerical scales and high-dimensional spatial complexity before fusion.

[0050] Specifically, the second tensor can generate a thirteenth gating weight using an activation function (such as Sigmoid). This thirteenth gating weight is dynamically calculated based on the current input inertia data. For example, the thirteenth gating weight = Sigmoid(Linear(twelfth tensor)).

[0051] In this embodiment, the purpose of convolution in the preset direction is to capture local temporal patterns in inertial data, such as the acceleration change trend within a recent predetermined time (a few milliseconds). Dynamic filtering is achieved by element-wise multiplying the calculated gating weights (the thirteenth gating weight) with the main feature after convolution (the convolution result of the eleventh tensor). This means that important and anomalous motion features are amplified (gating value ~1), while irrelevant or noisy data is suppressed (gating value ~0). Therefore, the thirteenth feature is the high-quality inertial feature after local temporal feature extraction and dynamic gating filtering. This processing ensures that only significant motion information is fed into the subsequent complex SSM for long-range modeling, greatly improving the model's efficiency and robustness.

[0052] In some embodiments, when the third network is a network structure with multi-head self-attention, the acquisition of the first fusion feature includes: spatially embedding the first visual feature and the first inertial feature within the same group; applying spatially embedded features to the first visual feature within the same group to obtain the corresponding spatial query vector; applying spatially embedded features to the first inertial feature within the same group to obtain the corresponding key name and value vector; obtaining an attention score based on the spatial query vector and key name; obtaining a first weighted value output based on the attention score and value vector; obtaining a feature vector within the first group based on the spatial query vector, key name, and value vector; and concatenating and normalizing the feature vectors within the first group output by multiple attention heads to obtain the first fusion feature.

[0053] In this embodiment, the attention score is used to measure the similarity between the spatial query vector and the value vector of each key. Specifically, the attention score can be obtained by performing a dot product on the spatial query vector and the value vector; a larger dot product indicates that the directions are closer and the angle between them is smaller.

[0054] Specifically, obtaining the weighted output is a weighted summation process, which involves multiplying each attention score by its corresponding value vector and summing all the results. For example, suppose there are n value vectors (v1, v2, ..., vn), each vector has a dimension of d_v, and n corresponding attention weights. The first weighted output S is: .

[0055] In some embodiments, the process of obtaining global fusion features includes: compressing the second fusion features of each group to a unified group dimension to obtain compressed features corresponding to the group; and concatenating the compressed features according to the channel dimension to obtain global fusion features.

[0056] Since different groups generate their own second fusion features, which may be vectors or small feature maps, but whose sizes or dimensions are not the same—for example, the network structures for processing the arm group and the torso group may be different, resulting in different output dimensions—these features can be compressed to the same dimension through aggregation operations. Specifically, this can be achieved using a global average pooling layer, which calculates the average of the second fusion features for each group; or by using a fully connected layer to map the second fusion features of each group to a fixed dimension.

[0057] In some embodiments, the process of obtaining the first unfolded feature includes: expanding the global fusion feature according to the total number of joints to obtain the first unfolded feature. The expansion process provides each joint with a complete first unfolded feature, providing sufficient information for accurate subsequent localization. Specifically, expansion can be achieved through duplication. For example, assuming the global fusion feature is a vector of dimension D, the global fusion feature is copied a total number of times (e.g., if the total number of joints is 17, then the global fusion feature is copied 17 times), stacked into a new matrix, and then output, thus obtaining the first unfolded feature. In application, during the duplication process, a joint identity identifier can also be added to each joint. This provides a global context for each joint and injects unique information for each joint, helping the network better distinguish the tasks of different joints and ensuring that the joint positions for subsequent human pose estimation are physically reasonable.

[0058] In some embodiments, when the third network is a network structure with a cross Mamba, the acquisition of the first fusion feature includes: mapping the first inertial feature to a global feature to obtain a first inertial mapping feature; extracting a first temporal feature after linear transformation of the first visual feature in the same group; performing a linear transformation on the first inertial mapping feature in the same group and extracting a second temporal feature; obtaining a first preliminary fusion feature and a second preliminary fusion feature based on the first temporal feature and the second temporal feature; and obtaining the first fusion feature based at least on the first preliminary fusion feature and the second preliminary fusion feature.

[0059] According to one embodiment of this application, human pose estimation is obtained using a regression head processing global model, which includes a layer normalization layer and a linear layer.

[0060] Specifically, inertial features can be mapped to a new feature space that is more compatible with the visual feature space through a linear projection layer (fully connected layer), thereby simplifying the subsequent fusion process and simultaneously serving to reduce dimensionality and extract core information. A linear transformation is used to project visual and inertial features onto the Cross-Mamba network structure, allowing the Cross-Mamba network structure to operate along the time dimension. This efficiently captures the temporal dependencies within these two modalities, resulting in visual and inertial features rich in their own modal temporal context information.

[0061] Specifically, a preset gating function (such as sigmoid) can be used to process the first and second time-domain features. For example, the first preliminary fusion feature is obtained by activating the second time-domain feature based on the first time-domain feature, and the second preliminary fusion feature is obtained by activating the first time-domain feature based on the second time-domain feature. That is, a feature that is mainly visual and supplemented by inertial information is obtained, and another feature that is mainly inertial and supplemented by visual information is obtained. Refined information interaction is achieved through cross-gating.

[0062] Specifically, the first preliminary fusion feature can be obtained by splicing the first preliminary fusion feature and the second preliminary fusion feature, or by summing the first preliminary fusion feature and the second preliminary fusion feature.

[0063] In some embodiments, the method further includes: aligning the visual data and the inertial measurement data after acquisition, so as to process the aligned visual data and the inertial measurement data.

[0064] Specifically, alignment processing includes time alignment and spatial alignment.

[0065] Specifically, time alignment includes timestamp-based downsampling alignment, or interpolation or padding based on a fixed time window. Time alignment synchronizes the visual acquisition device and the inertial measurement device, ensuring correspondence between visual and inertial measurement data. This prevents the fused data from being affected by excessive noise due to asynchrony between the two types of data, thus avoiding negative impacts on subsequent estimations.

[0066] In practical implementation, the time point of the device with a low sampling frequency (usually a camera, i.e., a visual acquisition device) can be used as a reference. The data point in the inertial measurement data that is closest to the camera frame timestamp is selected, thus downsampling the inertial measurement data to the same frequency as the visual data. In application, a fixed-length time window can be defined (e.g., 33.3 milliseconds, corresponding to the camera's frame interval). All inertial measurement data within the time window is collected for each frame, and then processed using interpolation or padding methods. For example, if padding is used, all inertial measurement data within the time window can be treated as a data block or sequence and paired with each frame. In application, the time window can be symmetrically centered on the timestamp of each frame, i.e., 16.65 milliseconds before and after the timestamp, for a total window of 33.3 milliseconds. This ensures that the inertial measurement data better covers the motion information before and after the image capture moment. This padding method fully utilizes the inertial measurement data, avoids information loss, and eliminates the need for complex interpolation calculations.

[0067] Specifically, spatial alignment is based on rotation matrices from the IMU's local coordinate system to the skeletal reference coordinate system, from the skeletal reference coordinate system to the sensor coordinate system, from the global coordinate system to the camera coordinate system, and from the sensor coordinate system to the global coordinate system. Spatial alignment synchronizes all sensors (such as visual acquisition devices and inertial measurement units) to the same coordinate system, enabling corresponding data processing and fusion of data from different modalities. The skeletal reference coordinate system represents a coordinate system with a fixed origin at a specific joint point on the human body (e.g., the shoulder), and its direction is fixed relative to the body's orientation. The sensor coordinate system represents a coordinate system with the IMU sensor itself as its origin. The global coordinate system represents a fixed world coordinate system, such as a corner of a room. The camera coordinate system represents a coordinate system with the camera's optical center as its origin, and the Z-axis is typically the camera's optical axis. For example, suppose an IMU sensor is worn on the wrist of the user being tested, and the camera takes a picture from the side. First, the raw data output by the IMU (including acceleration and angular velocity) is read, and the rotation matrix from the sensor coordinate system to the global coordinate system is applied to estimate the rotation of the raw data output by the IMU in the global coordinate system. Then, the rotation matrix from the skeletal reference coordinate system to the sensor coordinate system is applied to calibrate the sensor wearing position. Next, the rotation matrix from the skeletal reference coordinate system to the global coordinate system is obtained by combining the previous results to obtain the skeletal orientation (e.g., the orientation of the wrist relative to the room floor). Then, the camera is calibrated by applying the rotation matrix from the global coordinate system to the camera coordinate system to obtain the camera's shooting position (e.g., the camera shooting the room from the side). Finally, all transformations are concatenated to obtain the rotation matrix from the skeletal reference coordinate system to the camera coordinate system, calibrating the orientation of the wrist bones relative to the camera.

[0068] This application transforms the raw rotation data measured by the IMU based on its own coordinate system into skeleton rotation information based on the camera coordinate system through a series of spatial alignment transformations. In other words, it synchronizes multimodal data to the same coordinate system, reduces the learning difficulty of subsequent networks, and provides a foundation for improving the accuracy of subsequent multimodal data fusion.

[0069] The following is combined with Figure 2 The PoseVIMamba algorithm model shown herein provides a detailed description of this application. The input to this algorithm model is 2D pose keypoints extracted from RGB video. The quaternions were synchronously acquired and spatiotemporally aligned by M inertial measurement units (IMUs) deployed on the torso and limbs. The two together form a time window of length T. The output of this algorithm framework is the relative 3D pose coordinates of the J root joints contained in frame T. According to... Figure 2As can be seen, the algorithm model provided in this application mainly includes: a data preprocessing stage, a feature extraction stage, a multi-modal feature fusion stage, and a pose regression stage.

[0070] Data preprocessing mainly includes temporal alignment and spatial alignment. Regarding temporal alignment, since the IMU's sampling rate is higher than the camera's, a precise synchronization strategy is required. To achieve this, this embodiment performs downsampling based on timestamps and then uses interpolation techniques within a sliding window. Regarding temporal alignment... Visual data within the frame sliding window Extracted from monocular RGB video The keypoints are composed of pixel coordinates. These keypoints are detected using a human detector (such as YOLOv3) and a 2D pose estimator (such as MediaPipe or SimpleNet). To ensure scale invariance between frames, the keypoint coordinates are normalized to the range [-1, 1]. Simultaneously, spatial calibration is used to align the inertial data. This data consists of quaternions from IMUs placed in the pelvis, sternum, and limbs. The measurements represent rotations in the local coordinate system. To integrate these measurements into a unified skeleton-based coordinate system, a calibration process is required, using the following formula: ,in Indicates the measurement direction of the IMU in its local coordinate system. This indicates the relative rotation between the IMU and the corresponding skeleton. Transform the IMU's local coordinate system to a global (e.g., Vicon) coordinate system; and All results were obtained through a standardized T-pose calibration procedure. Furthermore, Camera extrinsic parameters were taken into account, and the global coordinate system was mapped to the camera coordinate system.

[0071] The feature extraction stage employs the Mamba model, known for its efficiency in long sequence modeling, as the feature extractor. For example... Figure 2 As shown, the feature extraction stage is divided into a visual branch and an inertial branch to process each modality independently. In the visual branch, 2D keypoint coordinates... Projecting the linear layer onto a high-dimensional feature space To obtain rich visual features In the inertial branch, the 2D key points and IMUs are grouped according to Table 1. The inertial measurement units (IMUs) are divided into five groups, each containing... There are 10 IMUs. Therefore, in each group, because the number of visual keypoints does not correspond to the number of IMUs, such as... Figure 3 As shown, taking the torso as an example, there are 5 visual points ( , , , , ), while there are only 2 IMUs ( , Since feature fusion (concatenation) cannot be directly performed, this application employs a Map layer for adaptive mapping. This layer adaptively aligns the IMU features in each group with the visual keypoints within that group. The inertial data from these groups are also projected onto a unified embedding dimension to obtain inertial features. Alignment with the visual branch representation. This adaptive mapping using the Map layer has two effects: First, shallow adaptive alignment; after Map mapping, IMUs within the same group will establish a one-to-one mapping relationship with feature points, and the relationship weights will be adaptively optimized during model training, thus enabling the model to learn the importance relationship between IMUs and feature points; Second, deep feature fusion; after shallow adaptive alignment, deep feature tensors have the same dimension and can be concatenated and fused to further learn deeper fused features. For example, if the tensor shape of Y_kps(torso) is [T,5,64] and the tensor shape of Y_IMU(torso) is [T,2,64], they cannot be directly concatenated and fused. However, after Map mapping, the tensor shape of Y_IMU(torso) is transformed into [T,5,64]. In application, if the same number of IMUs as the visual keypoints are used, they can also be directly concatenated and fused.

[0072] The core of feature extraction is as follows Figure 2 The image shows the Self-STMamba module (Self-Spatio-TemporalMamba). The architecture of the Self-STMamba module is as follows: Figure 4 As shown, it mainly utilizes the Selective State Space (SSM) mechanism to capture dependencies in the spatial (joint) and temporal (frame) dimensions.

[0073] To preserve spatial details, learnable spatial embeddings are applied to the visual keypoint features and each set of IMU features before processing by the Self-STMamba module. (spatio position encoding), then layer normalization is performed. The processing algorithm formula is: , Presentation layer normalization operation, Indicates visual features or inertial characteristics Spatial location features are injected into spatial location coding. , Indicates grouping.

[0074] In the Self-STMamba module, input features are linearly projected onto the internal dimensions of the SSM to enhance the model's ability to capture complex spatiotemporal dependencies.

[0075] The projected tensor is as follows Figure 4 It was split into two components: and The projection algorithm formula is: ,in, This indicates a split or truncation operation. This represents the projection operation (linear mapping) when inputting the module. For Depth convolutions are applied to capture local patterns, followed by activation functions. (such as SiLU) to enhance local feature interactions; at the same time, It is activated and used as a gating mechanism to scale the SSM output.

[0076] In addition, the processing of visual and inertial features differs in the Self-STMamba module to adapt to modality-specific features.

[0077] For inertial features, VMamba's bidirectional global spatiotemporal scanning mechanism is utilized. This method employs four scanning directions—forward space, forward time, backward space, and backward time—allowing each IMU sample within the sliding window to integrate information from all other samples in the limbs and time steps.

[0078] For visual features, such as Figure 3 The parent joint relationships defined in the algorithm are reordered and then superimposed onto the scan sequence of the child joints. This scanning algorithm generates a tensor. It contains sequences derived from four scan paths, used to compute SSM parameters and construct spatiotemporal feature maps. The algorithm formula is: ,in, This indicates a cross-scan operation. This represents the convolution operation. The activation function (such as SiLU) is determined by the gating parameters. The modulated SSM output is projected back to the unified embedding dimension. Then, a multilayer perceptron (MLP) layer and residual connections are applied to generate the final output, which is formalized as follows: ; ; , This represents a multilayer perceptron, specifically two linear layers coupled with the GeLU activation function. This indicates that a linear projection was used for the output. For the output tensor... By applying temporal and positional embedding and employing another Self-STMamba module to further extract the spatiotemporal correlations between joints within the sliding window, a comprehensive modeling of the dynamic dependencies between the two modalities is ensured.

[0079] To achieve robust cross-modal information integration, the multimodal feature fusion module (i.e., the third network) integrates multimodal inputs through a standard multi-head cross-attention mechanism. For example... Figure 5 As shown, visual features are extracted into different groups. In the middle. For each group Visual features are projected as "queries". inertial characteristics Generate "keys" and "value" .

[0080] The cross-attention output for each head is calculated as follows:

[0081] ,in The dimension representing the key, i.e., the embedding dimension. h multi-head attention heads (typically set to h=8) are computed in parallel, and then the outputs of each attention head are concatenated and normalized. It is worth noting that residual connections are applied only to visual features to preserve their structural integrity.

[0082] After the initial fusion, such as Figure 2 use A bidirectional spatiotemporal Self-STMamba module extracts deeper representations from the fused features. Grouped fused features. First, it is compressed to a unified group dimension. Then, the features from all groups are concatenated to form a globally fused feature. Then the fusion features of each frame are expanded to To capture comprehensive global dependencies, the skeleton-aware Self-STMamba module, which stacks depth layers, is applied for final global modeling.

[0083] In the pose regression stage, the regression head aims to generate a final 3D root relative pose prediction for the entire sequence. It consists of a layer normalization layer and a linear layer, which processes the fused multimodal features to generate the output pose. . represents the relative 3D coordinates of the J root joints contained in frame T, as the final prediction result of the 3D human pose estimation task.

[0084] Unlike models based on GCN, CNN, or Transformer, which require retraining for different input sequence lengths, this model... Figure 2 The model architecture used in this application natively supports variable-length sequences. Therefore, during training and inference, it can handle input sequences of arbitrary lengths under maximum constraints without requiring architectural changes, greatly enhancing flexibility. To leverage this, this application sets the maximum sequence length T=81 and randomly samples from sequence parameters of length {9,18,27,36,45,54,63,72,81} in each batch during training, exposing the model to different temporal resolutions. Testing was conducted on personal consumer hardware (RTX 4060 Laptop) using fixed-length sequence inputs, and the stability of the training strategy was verified through ablation studies.

[0085] The PoseVIMamba algorithm model used in this application is implemented based on the PyTorch framework and trained on a single NVIDIA RTX3090 GPU. During training, the batch size was configured to 16 sequences, and the AdamW optimizer was used to optimize the network parameters over 15 epochs, with weight decay set to 0.01. The learning rate was initialized to... It follows an exponential decay strategy with a decay factor of 0.99. The total loss function weights are set to... , , , The network hyperparameters are set as follows: Embedding dimension Group Dimension The ablation experiment results for these configurations are shown in Table 5. The number of Self-STMamba blocks in each group... And the number of stacked skeletal-aware Self-STMamba layers. .

[0086] P1 and P2 were used as metrics, where P1 is the Mean Per-JointPosition Error (MPJPE) and P2 is the Procrustes-aligned MPJPE. The algorithm model PoseVIMamba, namely the Mamba-Cross-Attention hybrid model, used in this application was compared with the RNN-based method of Pan et al., the Transformer-based method of Wang et al., and the CNN-based method of Bao et al. on the TotalCapture dataset to evaluate PoseVIMamba, as shown in Table 1.

[0087] Table 1 - Comparison of Model Evaluation Indicators

[0088]

[0089] As can be seen, using MediaPipe (mp) and 6 IMUs, PoseVIMamba achieves an average MPJPE of 33.2 mm and a P-MPJPE of 26.7 mm, outperforming Wang et al.'s method (39.0 mm, 28.8 mm) and Pan et al.'s method (48.1 mm, 32.9 mm). Using 6 IMUs and SimpleNet (sn) as the 2D detector, our model achieves an MPJPE of 33.1 mm and a P-MPJPE of 24.1 mm, surpassing Wang et al.'s method (34.9 mm, 26.9 mm). Using 8 IMUs and SimpleNet, PoseVIMamba achieves an MPJPE of 32.8 mm and a P-MPJPE of 23.7 mm, outperforming Wang et al.'s method (33.4 mm, 25.1 mm) and Bao et al.'s method (33.7 mm).

[0090] Therefore, using a GT with 6 IMUs as input, PoseVIMamba achieved the lowest errors (MPJPE: 17.2 mm, P-MPJPE: 13.8 mm), which are improvements of 11.4 mm and 3.8 mm, respectively, over Wang et al.'s method (28.6 mm, 17.6 mm). Overall, PoseVIMamba achieves state-of-the-art performance with an average MPJPE of 17.2 mm (GT) and approximately 33 mm (2D detector), demonstrating strong accuracy and generalization across different subjects and actions.

[0091] The stability of the variable-length input training strategy is evaluated below with reference to Table 2. Experiments used real 2D keypoints of fixed length (9, 27, and 81 frames) and dynamic length as input. The results in Table 2 show that using variable length (V) as input achieved an average MPJPE (Avg) of 18.9 mm, close to the best performance with fixed length (17.2 mm at T=81), and exhibited minimal variation in accuracy across sequence lengths (e.g., 21.1 mm at T=9 and 17.2 mm at T=81). These results demonstrate that the variable-length training method maintains stable performance and provides greater flexibility for model application by supporting different sequence length inputs.

[0092] Table 2 - Performance Analysis Comparison of Variable-Length Training Strategies

[0093]

[0094] To evaluate the hybrid architecture after model variants, the following approach is adopted: Figure 6The Cross-Mamba module replacement shown Figure 2 The Cross-Attention module is used in the Cross-Mamba module. First, a linear layer is used to establish an adaptive mapping between inertial and visual features within the same group. The relational weights are adaptively optimized during model training, allowing the model to learn the importance relationship between the IMU and feature points within the group. Similar to the Cross-Attention module, the input visual or inertial features are projected using two linear layers respectively to enhance the model's ability to capture complex spatiotemporal dependencies. Then, one-dimensional convolution (Conv1D) is applied to the visual and inertial features respectively to enhance their respective local spatiotemporal pattern representation. Unlike the four-way scan in the Self-STMamba module, the Cross-Mamba module connects the visual and inertial modalities along the spatial axis and uses bidirectional scans (forward and backward) to maintain comparability with the Cross-Attention module. The fused features are then re-decomposed into visual components. and inertial components and their respective activation functions The SSM output is scaled by multiplying the mapping features (such as SiLU). Finally, the visual and inertial features are concatenated and mapped back to a unified group dimension. Obtain group fusion features .

[0095] Based on the comparative analysis provided in Table 3 with the same number of IMUs and hyperparameter settings, the results show that the Cross-Attention-based method consistently outperforms the Cross-Mamba-based method, especially on unseen subject test sets such as UFS3 and UA3, where MPJPE is reduced by more than 5.6 mm. This performance gap highlights that the Mamba-Cross-Attention hybrid framework proposed in this application effectively overcomes the limitations of the pure Mamba architecture in multimodal fusion modeling of complex spatial dependencies, and improves the performance of the vision-inertial fusion-based 3DHPE model.

[0096] Table 3 - Performance Comparison of Hybrid Architectures for Model Variants

[0097]

[0098] In the aforementioned comparative and model variant studies, the main focus was on setting the model embedding dimension. Group Dimension and the size of the state space Given that the purpose of this application is to develop an algorithmic framework that balances real-time performance and accuracy, a series of experiments were conducted to further explore the combined impact of these three parameters on model accuracy (using MPJPE as the evaluation metric), parameter count, and running speed (FPS). MPJPE and FPS were tested on the entire test set using MediaPipe to detect 2D poses at a fixed time length T=27 as input, and the results are shown in Table 4.

[0099] Analysis of experiments 1 to 4 shows that... It has a significant impact on reasoning speed (FPS): when When the state space dimension was reduced from 16 to 1, the FPS increased from 61.4 to 69.4, indicating that reducing the state space dimension can effectively improve the model's running efficiency. On the other hand, the embedding dimension... Group Dimension It significantly affects the number of model parameters. When fixed At that time, The dimension increased from 64 to 96, and the number of parameters increased by 1.4M, but the accuracy not only did not improve but slightly decreased. And when fixed... And When the group dimensions were reduced from 256 to 128 and 64, the number of parameters decreased from 12.3M to 3.9M and 1.8M, respectively, and the FPS also increased significantly from 61.4 to over 70, indicating that reducing the group dimensions... It can significantly reduce model size and computational burden.

[0100] Regarding model accuracy, MPJPE increases with decreasing dimensions, indicating that dimensionality reduction leads to accuracy loss. For example, when... and When fixed at 64 and 256, Decreasing from 16 to 1 results in an increase in MPJPE from 33.2 to 35.2. When Fixed at 64, When fixed at 16, The MPJPE values ​​decreased from 256 to 128 and 64, respectively, while the MPJPE values ​​increased from 33.1 to 34.5 and 36.7. Experiment 6 showed that although there was a slight decrease in accuracy compared to Experiment 1, reducing the group dimension... The number of parameters has been significantly reduced and the inference speed has been improved, making it highly cost-effective for deployment on resource-constrained edge devices. While No. 8 and No. 9 have faster inference speeds than No. 6, they have a larger number of parameters and lower accuracy.

[0101] In conclusion, while reducing dimensionality can improve computational efficiency and reduce model size, its negative impact on accuracy must be weighed.

[0102] Table 4 - Comparison of Model Accuracy Parameters

[0103]

[0104] The model performance analysis in Table 5 reveals that all Mamba-based models achieved real-time inference capabilities exceeding 60 FPS (standard RGB camera sampling rate), while the Transformer-based model (45.8 FPS) showed a slight latency. Although the Cross-Mamba-based model had the largest number of parameters, the memory usage of the model on the device was determined by memory I / O during inference, not simply by the number of parameters. As the table shows, the Cross-Mamba-based model had the lowest memory peak during runtime, while the GCN-Transformer-based model had the highest, indicating that the number of parameters is not a key factor in memory and computational efficiency.

[0105] Compared to GCN-Transformerd-based models, PoseVIMamba (Cross-Attention-B) achieves faster inference speed while demonstrating more competitive memory efficiency. However, the comparison between Cross-Mamba and Cross-Attention-B reveals a key trade-off: while Cross-Mamba excels in unimodal feature extraction due to its efficient sequence modeling, its cross-modal fusion is less effective than Cross-Attention in complex scenarios, as demonstrated in Table 3. Despite achieving faster inference speed (64.6 FPS vs. 61.4 FPS), Cross-Attention-B exhibits a lower average MPJPE (33.1 mm vs. 35.3 mm).

[0106] Table 5 - Model Performance Comparison

[0107]

[0108] According to another embodiment of this application, a human pose estimation device based on visual-inertial fusion is provided, such as... Figure 7 As shown, the device 70 includes: a visual data processing module 701, an inertial data processing module 702, a first fusion processing module 703, a second fusion processing module 704, a global fusion processing module 705, a global feature expansion module 706, a global model construction module 707, and a human pose estimation module 708.

[0109] The visual data processing module 701 is used to process visual data using a first network to obtain first visual features;

[0110] The inertial data processing module 702 is used to process inertial measurement data using a second network to obtain a first inertial feature;

[0111] The first fusion processing module 703 is used to obtain the first fused feature by fusing the first visual feature and the first inertial feature based on the third network;

[0112] The second fusion processing module 704 is used to process the first fusion feature using the fourth network to obtain the second fusion feature;

[0113] The global fusion processing module 705 is used to obtain global fusion features based on the second fusion features corresponding to multiple groups;

[0114] The global feature extension module 706 is used to extend the global fused features to obtain the first expanded features;

[0115] The global model building module 707 is used to build a global model from the first unfolded features based on the fifth network with stacked deep layers, thereby obtaining a global model.

[0116] The human pose estimation module 708 is used to obtain an estimate of human pose based on a global model.

[0117] The first, second, fourth, and fifth networks have a self-attention spatiotemporal Mamba network structure;

[0118] The third network has a network structure with multi-head self-attention or cross-mamba.

[0119] In this embodiment, feature extraction networks are designed for visual data and inertial measurement data respectively. The original data is transformed into first visual features and first inertial features. Then, based on a third network, the first visual features and the first inertial features are fused to obtain a first fused feature. This feature extraction method using a dedicated network can remove redundant and interfering information unrelated to human pose estimation from heterogeneous original data. On the other hand, it can reduce the data dimensionality, reduce the computational overhead of fusion, and avoid noise interference in the fusion process. At the same time, the use of a fourth network makes the output second fused feature smoother and more reliable. Thus, the global fused feature obtained by grouping the second fused features into multiple groups contains all the information of human pose. A global model is constructed based on the expanded first features, which can represent human pose and solve the problem of poor human pose estimation accuracy caused by data inaccuracy due to environmental influences and noise.

[0120] According to one embodiment of this application, visual data and inertial measurement data are time-aligned, including timestamp-based downsampling alignment or interpolation or padding based on a fixed time window.

[0121] According to one embodiment of this application, visual data and inertial measurement data are spatially aligned. The spatial alignment is based on the rotation matrix from the local coordinate system of the inertial measurement unit to the skeletal reference coordinate system, the rotation matrix from the skeletal reference coordinate system to the sensor coordinate system, the rotation matrix from the global coordinate system to the camera coordinate system, and the rotation matrix from the sensor coordinate system to the global coordinate system.

[0122] According to one embodiment of this application, the first visual feature is obtained as follows:

[0123] Spatial location feature injection is applied to the visual key point features contained in the first visual feature, followed by layer normalization to obtain the first input feature;

[0124] The first input feature is linearly projected onto the state-space model to obtain the first tensor and the second tensor.

[0125] The second feature is obtained by performing convolution and gating on the first tensor.

[0126] The second feature is expanded to obtain the third tensor based on the rescanning and sorting of the second feature and the parent joint feature;

[0127] The third tensor is used as the input to the state-space model, and the fourth tensor is obtained based on the second gating function and the second tensor.

[0128] The fifth tensor is obtained based on the linear transformation of the first input features and the fourth tensor;

[0129] Apply layer linear normalization to the fifth tensor, and use the normalized tensor as the input to the fully connected network to obtain the sixth tensor;

[0130] The first visual feature is obtained based on the fifth and sixth tensors.

[0131] According to one embodiment of this application, the first inertial feature is obtained as follows:

[0132] The inertial measurement units are grouped to have the same embedding dimension as the visual branch;

[0133] Spatial position feature injection is applied to the inertial measurement data contained in the first inertial feature, followed by layer normalization to obtain the second input feature;

[0134] The second input feature is linearly projected onto the state-space model to obtain the eleventh and twelfth tensors;

[0135] The thirteenth gating weight is generated based on the twelfth tensor;

[0136] The thirteenth feature is obtained based on the convolution result of the thirteenth gate weight and the eleventh tensor in a preset direction;

[0137] The thirteenth tensor is used as the input to the state-space model, and the fourteenth tensor is obtained based on the second gating function;

[0138] The fifteenth tensor is obtained by linear transformation of the eleventh input feature and the fourteenth tensor;

[0139] Apply layer linear normalization to the fifteenth tensor and use the normalized tensor as the input to the fully connected network to obtain the sixteenth tensor;

[0140] The first inertial feature is obtained based on the fifteenth and sixteenth tensors.

[0141] According to one embodiment of this application, the third network has a multi-head self-attention network structure;

[0142] The first fusion feature is obtained as follows:

[0143] Spatial location embedding is performed on the first visual feature and the first inertial feature within the same group.

[0144] Spatial location embedding is applied to the first visual feature within the same group to obtain the corresponding spatial query vector;

[0145] Spatial location embedding is applied to the first inertial feature within the same group to obtain the corresponding key name and value vector;

[0146] Attention scores are obtained based on spatial query vectors and key names;

[0147] The output is obtained by adding the first weighted value based on the attention score and the value vector;

[0148] The feature vector within the first group is obtained based on the spatial query vector, key name, and value vector.

[0149] The first set of intra-feature vectors output by multiple attention heads are concatenated and normalized to obtain the first fused feature.

[0150] According to one embodiment of this application, the process of obtaining global fusion features includes:

[0151] The second fusion feature of each group is compressed to a unified group dimension to obtain the compressed feature corresponding to the group.

[0152] The compressed features are concatenated along the channel dimension to obtain the global fused features.

[0153] According to one embodiment of this application, the process of obtaining the first unfolded feature includes:

[0154] The global fusion features are expanded according to the total number of joints to obtain the first expanded features.

[0155] According to one embodiment of this application, the third network has a cross-Mamba network structure;

[0156] The first fusion feature is obtained as follows:

[0157] Map the first inertial feature to the global feature to obtain the first inertial mapping feature;

[0158] The first temporal feature is extracted by performing a linear transformation on the first visual feature within the same group, and the second temporal feature is extracted by performing a linear transformation on the first inertial mapping feature within the same group.

[0159] First preliminary fusion features and second preliminary fusion features are obtained based on first time-domain features and second time-domain features;

[0160] At least the first preliminary fusion feature and the second preliminary fusion feature are used to obtain the first fusion feature.

[0161] According to one embodiment of this application, human pose estimation is obtained using a regression head processing global model, which includes a layer normalization layer and a linear layer.

[0162] The human pose estimation device based on visual-inertial fusion in this embodiment can execute the human pose estimation method based on visual-inertial fusion shown in the embodiment of this application. The implementation principle is similar and will not be described again here.

[0163] Another embodiment of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0164] Specifically, the processor can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0165] Specifically, the processor connects to the memory via a bus, which may include a path for transmitting information. The bus can be a PCI bus or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0166] The memory may be ROM or other types of static storage devices that can store static information and instructions, RAM or other types of dynamic storage devices that can store information and instructions, or EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0167] Optionally, the memory stores the code of a computer program that executes the scheme of this application, and the execution is controlled by a processor. The processor executes the application code stored in the memory to implement the operation of the apparatus provided in the above embodiments.

[0168] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions for performing the methods provided in the above embodiments.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0171] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A human pose estimation method based on visual-inertial fusion, characterized in that, include: The first network is used to process visual data to obtain first visual features; The inertial measurement data is processed using a second network to obtain the first inertial feature; The first fused feature is obtained by fusing the first visual feature and the first inertial feature based on the third network; The first fusion feature is processed using a fourth network to obtain the second fusion feature; Global fusion features are obtained based on the second fusion features corresponding to multiple groups; the multiple groups include five groups: left arm, right arm, left leg, right leg, and torso. The global fusion features are expanded to obtain the first expanded features; The fifth network, based on stacked deep layers, constructs a global model from the first expanded features to obtain the global model. Human pose estimation is obtained based on a global model; The first network, the second network, the fourth network, and the fifth network have a self-attention spatiotemporal Mamba network structure; The self-attention spatiotemporal Mamba network structure adopts the Self-STMamba module, which mainly utilizes the Selective State Space (SSM) mechanism to capture dependencies in the spatial and temporal dimensions. Before processing by the Self-STMamba module, learnable spatial location embeddings are applied to the visual keypoint features and each set of IMU features, followed by layer normalization. In the Self-STMamba module, the input features are linearly projected onto the internal dimensions of the SSM; the projected tensor is then split into two components. and ;for The algorithm applies depthwise convolutions to capture local patterns, then uses activation functions to enhance local feature interactions before inputting into SS2D. This data is then processed sequentially through cross-scanning units, selective scanning units, and cross-fusion units before being output. Simultaneously... It is activated and used as a gating mechanism to scale the SSM output, determined by the gating parameters. The modulated SSM output is projected back to the unified embedding dimension, and then a multilayer perceptron (MLP) layer and residual connections are applied to generate the final output. The third network has a multi-head self-attention or cross-Mamba network structure; The Cross-Mamba network structure employs a Cross-Mamba module. The Cross-Mamba module first uses a linear layer to establish an adaptive mapping between inertial and visual features within the same group. The relational weights are adaptively optimized during model training, enabling the model to learn the importance relationship between the IMU and feature points within the group. The input visual or inertial features are projected using two linear layers respectively to enhance the model's ability to capture complex spatiotemporal dependencies. Then, one-dimensional convolutions are applied to the visual and inertial features respectively to enhance their respective local spatiotemporal pattern expressiveness. The Cross-Mamba module connects the visual and inertial modalities along the spatial axis and uses bidirectional scanning to maintain comparability with the Cross Attention module. The fused features are re-split into visual and inertial components and multiplied with their respective activated mapping features to scale the SSM output. Finally, the visual and inertial features are concatenated and mapped back to a unified group dimension to obtain the group fused features.

2. The method as described in claim 1, characterized in that, The first visual feature is obtained in the following manner: Spatial location features are injected into the visual keypoint features contained in the visual data, followed by layer normalization to obtain the first input features; The first input feature is linearly projected onto the state-space model to obtain the first tensor and the second tensor; The second feature is obtained by performing convolution and gating on the first tensor. The second feature is expanded to obtain the third tensor based on the rescanning and sorting of the second feature and the parent joint feature; The third tensor is used as the input to the state-space model, and the fourth tensor is obtained based on the second gating function and the second tensor. The fifth tensor is obtained based on the linear transformation of the first input features and the fourth tensor; Apply layer normalization to the fifth tensor and use the normalized tensor as input to the fully connected network to obtain the sixth tensor; The first visual feature is obtained based on the fifth and sixth tensors.

3. The method as described in claim 1, characterized in that, The first inertial feature is obtained in the following manner: The inertial measurement units are grouped to have the same embedding dimension as the visual branch; Spatial position feature injection is applied to the inertial measurement data contained in the first inertial feature, followed by layer normalization to obtain the second input feature; The second input feature is linearly projected onto the state-space model to obtain the eleventh and twelfth tensors; The thirteenth gating weight is generated based on the twelfth tensor; The thirteenth tensor is obtained based on the convolution result of the thirteenth gating weight and the eleventh tensor in a preset direction; The thirteenth tensor is used as the input to the state-space model, and the fourteenth tensor is obtained based on the second gating function; The fifteenth tensor is obtained by linear transformation of the eleventh and fourteenth tensors; Apply layer normalization to the fifteenth tensor and use the normalized tensor as input to the fully connected network to obtain the sixteenth tensor; The first inertial feature is obtained based on the fifteenth and sixteenth tensors.

4. The method as described in claim 1, characterized in that, When the third network has a multi-head self-attention network structure, the first fusion feature is obtained in the following manner: Spatial location embedding is performed on the first visual feature and the first inertial feature within the same group. Spatial location embedding is applied to the first visual feature within the same group to obtain the corresponding spatial query vector; Spatial location embedding is applied to the first inertial feature within the same group to obtain the corresponding key name and value vector; Attention scores are obtained based on spatial query vectors and key names; The first weighted output is obtained based on the attention score and the value vector; The feature vector within the first group is obtained based on the spatial query vector, key name, and value vector. The first set of intra-feature vectors output by multiple attention heads are concatenated and normalized to obtain the first fused feature.

5. The method as described in claim 1, characterized in that, The process of obtaining the global fusion features includes: The second fusion feature of each group is compressed to a unified group dimension to obtain the compressed feature corresponding to the group. The compressed features are concatenated along the channel dimension to obtain the global fused features.

6. The method as described in claim 1, characterized in that, The process of obtaining the first expanded feature includes: The global fusion features are expanded according to the total number of joints to obtain the first expanded features.

7. The method as described in claim 1, characterized in that, When the third network has a cross-Mamba network structure, the first fusion feature is obtained as follows: Map the first inertial feature to the global feature to obtain the first inertial mapping feature; The first temporal feature is extracted by performing a linear transformation on the first visual feature within the same group, and the second temporal feature is extracted by performing a linear transformation on the first inertial mapping feature within the same group. First preliminary fusion features and second preliminary fusion features are obtained based on first time-domain features and second time-domain features; At least the first preliminary fusion feature and the second preliminary fusion feature are used to obtain the first fusion feature.

8. The method as described in claim 1, characterized in that, The estimation of human posture is obtained by processing a global model using a regression head, which includes a normalization layer and a linear layer.

9. The method as described in claim 1, characterized in that, The method further includes: performing alignment processing after acquiring the visual data and the inertial measurement data, so as to process the aligned visual data and the inertial measurement data.

10. A human pose estimation device based on visual-inertial fusion, the device being used to implement the method as described in any one of claims 1 to 9, characterized in that, include: A visual data processing module is used to process visual data using a first network to obtain first visual features; An inertial data processing module is used to process inertial measurement data using a second network to obtain a first inertial feature; The first fusion processing module is used to obtain the first fused feature by fusing the first visual feature and the first inertial feature based on the third network; The second fusion processing module is used to process the first fusion feature using the fourth network to obtain the second fusion feature; The global fusion processing module is used to obtain global fusion features based on the second fusion features corresponding to multiple groups; The global feature extension module is used to extend the global fused features to obtain the first expanded features; The global model building module is used to build a global model from the first expanded features based on the fifth network with stacked deep layers, thereby obtaining the global model. The human pose estimation module is used to obtain an estimate of human pose based on a global model. The first network, the second network, the fourth network, and the fifth network have a self-attention spatiotemporal Mamba network structure; The third network has a multi-head self-attention or cross-mamba network structure.

Images