A method for full-body pose estimation based on sparse inertial sensors

Abstract

The application discloses a whole body posture estimation method based on a sparse inertial sensor. The application comprises the following steps: 1: synthesizing acceleration data, direction data and angular velocity data through a finite difference method and a forward kinematics algorithm; 2: performing alignment processing on original acceleration data and original direction data collected by three inertial sensors worn on the user's head joint, left wrist joint and right wrist joint through a rotation transformation matrix; 3: extracting deep features of inertial motion data samples by combining a Transformer Encoder and a bidirectional long short-term memory network encoder, designing a decoder based on a multilayer perceptron to calculate the positions and relative rotations of joints, and combining a human body kinematics model to estimate the relative rotations of joints in different parts in multiple stages to realize whole body posture estimation. The application can effectively overcome the defects of using sparse inertial input for whole body motion reconstruction in the existing posture estimation method, and effectively improve the robustness and prediction accuracy of the system.

Description

Technical Field

[0001] This invention belongs to the field of human motion capture technology, and particularly to posture estimation, specifically involving a whole-body posture estimation method based on sparse inertial sensors. Background Technology

[0002] The innovative fusion of virtual reality and motion capture can provide users with real-time feedback and high interactivity, ensuring accurate mapping and matching between real motion and virtual scenes, enabling users to participate more directly in the virtual environment, thereby achieving a seamless integration of reality and virtuality.

[0003] Vision-based motion capture applications are mature and hold a large market share. They utilize real-time image streams from multiple RGB cameras monitoring and tracking target feature points from different angles to achieve motion capture. This approach typically offers high real-time performance and responsiveness, but is susceptible to external environmental factors such as lighting conditions and obstructions. Marker-based optical systems are widely recognized for their accuracy. They work by using motion capture cameras to detect markers attached to human joints in real time, but require large installations and complex calibration procedures. With the rapid advancement of microelectromechanical systems (MEMS) technology, wearable sensors offer novel solutions for motion capture. Wearable sensor-based motion capture systems are not limited by space or environment, avoiding issues of monitoring area and environmental obstruction. The miniaturization of sensors also increases user comfort and makes them more suitable for mobile applications. However, commercial inertial motion capture systems like Xsens acquire joint acceleration, angular velocity, and magnetic field strength from 17 or more inertial sensors. Especially during extended use, the excessive number of sensors limits the diversity of motion and user comfort, and increases the management cost and computational complexity of multiple devices. Summary of the Invention

[0004] To overcome the limitations of existing technologies that rely solely on sparse inertial inputs from the head joint and left and right wrist joints for all whole-body posture estimations, this invention provides a whole-body posture estimation method based on sparse inertial sensors.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a whole-body attitude estimation method based on sparse inertial sensors, the specific identification steps of which are as follows:

[0006] Step S1: Synthesize the acceleration data, orientation data, and angular velocity data required for neural network training using the finite difference method and forward kinematics algorithm;

[0007] Step S2: Align the raw acceleration and orientation data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint using a rotation transformation matrix to transform them into a usable unified coordinate system.

[0008] Step S3: Extract depth features of inertial motion data samples by combining Transformer Encoder and bidirectional long short-term memory network encoder, design a multilayer perceptron-based decoder to calculate joint position and relative rotation, combine human kinematic model and estimate the relative rotation of joints in different parts in multiple stages to achieve whole-body posture estimation.

[0009] Furthermore, step S1 specifically includes the following steps:

[0010] S1.1: Place virtual inertial sensors on the mesh vertices corresponding to the head joint and the left and right wrist joints on the surface of the SMPL model, and use finite difference to calculate synthetic acceleration data at the same mesh position in adjacent time frames of the virtual inertial sensors:

[0011]

[0012] Where, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in frame t, i = 1, 2, 3, Δt is the time interval between two consecutive frames, and n is the smoothing factor;

[0013] S1.2: Using the forward kinematics algorithm, the synthetic orientation data of the three virtual inertial sensors is calculated from the real joint posture according to the SMPL model;

[0014] R i (t)=FK(p(t))

[0015] Where FK(·) is the forward kinematics algorithm, R i p(t) is the orientation data of the i-th virtual inertial sensor, and p(t) is the actual posture data of all joints in the body;

[0016] S1.3: Calculate the composite angular velocity data using the composite direction data:

[0017] W i (t)=R i (t-1) -1 R i (t)

[0018] Among them, W i (t) and R i(t) represents the synthesized direction data and synthesized angular velocity data of the i-th virtual inertial sensor in frame t, respectively.

[0019] Furthermore, step S2 specifically includes the following steps:

[0020] S2.1: A transformation matrix needs to be found to establish the global inertial coordinate system F. I and SMPL coordinate system F M The relationship between them is represented by R. MI :F I →F M Select any inertial sensor to wear on the head joint, ensuring that the sensor axis is aligned with the SMPL body frame, so that the sensor coordinate system F S and SMPL coordinate system F M It is identical; since the orientation data of the inertial sensor is represented by R. IS :F S →F I The orientation reading R of the inertial sensor at the head position Head The inverse matrix is ​​considered to be from F I To F M The transformation matrix R MI ;

[0021] S2.2: Wear the remaining two inertial sensors in any direction while maintaining a uniform T-shaped posture to obtain the coordinates relative to the sensor coordinate system F. S initial acceleration and relative to the global inertial coordinate system F I initial direction

[0022] The remaining two inertial sensors are worn on the left and right wrist joints in arbitrary orientations, while the body maintains a uniform T-shaped posture, in order to obtain data relative to the sensor coordinate system F. S initial acceleration and relative to the global inertial coordinate system F I initial direction Let R be the angular deviation between the sensor and the skeleton. BS If it is constant, then the skeleton's position relative to the global inertial coordinate system F is obtained. I initial direction

[0023] S2.3: Since the initial pose is uniform, the orientation of the entire skeleton can be calculated using the SMPL model. Therefore, the orientation of the skeleton in the SMPL coordinate system is... It is known that for any pose, the absolute orientation of the bones is equivalent in different coordinate systems, thus obtaining... Due to F M=F I R MI Therefore, the virtual skeleton orientation can be obtained, which is the orientation data of the inertial sensor in the skeleton coordinate system.

[0024] S2.4: For the raw acceleration data, first transform it to the global inertial coordinate system F based on the raw orientation data from the inertial sensor. I ,Right now Then acceleration data After subtracting gravity, transform to the SMPL coordinate system F. M The acceleration data from the inertial sensor in the SMPL coordinate system F M and skeletal coordinate system F B Since they are equal, they are simultaneously transformed to the skeletal coordinate system F. B .

[0025] Furthermore, step S3 specifically includes the following steps:

[0026] S3.1: By designing an encoder that combines a Transformer Encoder and a bidirectional long short-term memory network, the depth features of inertial data (including acceleration data, orientation data, and angular velocity data) are extracted. A decoder based on a multilayer perceptron is designed to calculate the position of the joints relative to the lumbar joint and their rotation relative to the parent joint. The human kinematics model is combined and the rotation of joints of different body parts relative to the parent joint is estimated in multiple stages. The inertial data includes acceleration data, orientation data, and angular velocity data.

[0027] S3.2: Combining a human kinematics model, the rotation of local key joints, upper body joints, and whole-body joints relative to the parent joint, as well as the global rotation of the lumbar joint, are estimated sequentially in multiple stages. Specifically: The first stage realizes the estimation from the inertial measurement value X to the local key joint pose p1 and the global orientation p. (1) The mapping process involves the position b of a local key joint as an intermediate step; the second stage estimates the upper body joint pose p2 and global orientation p from the inertial measurement value X and the local key joint pose p1 output from the first stage. (2) The third stage involves estimating the whole-body joint pose p3 and global orientation p3 from the inertial measurement value X and the upper-body joint pose p2 output from the second stage. (3) .

[0028] The technical effects and advantages of this invention are as follows:

[0029] This invention utilizes acceleration and orientation measurements from three inertial sensors—one on the user's head joint and the other on the left and right wrist joints—as input to design a whole-body posture estimation method based on sparse inertial sensors. First, it leverages the advantages of local inertial data to estimate the posture of key local joints. Then, it progressively extends to the core upper body areas such as the pelvis and torso, ultimately achieving accurate reconstruction of the whole-body motion. At the posture estimation level, a decoder combining a Transformer Encoder and a bidirectional Long Short-Term Memory (LSTM) network is specifically designed to improve the understanding of inertial motion information. A multilayer perceptron-based decoder shares rich and comprehensive high-dimensional posture features from the encoder, fully mapping the sparsely tracked acceleration and orientation measurements to the whole-body joints of the SMPL model. A sliding window-based testing strategy, conducive to real-time architecture deployment, is used, along with past and future motion information to predict the current posture, generating real-time animation demonstrations with acceptable latency. The proposed method effectively overcomes the shortcomings of existing posture estimation methods that use sparse inertial input for whole-body motion reconstruction, significantly improving the system's robustness and prediction accuracy. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0031] Figure 2 This is a schematic diagram of an inertial sensor wearable to which the present invention is applicable.

[0032] Figure 3 This is a schematic diagram of inertial sensor calibration and alignment.

[0033] Figure 4 The flowchart shows the whole-body pose estimation process.

[0034] Figure 5 An animated demonstration of real-time attitude estimation. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention provides, for example Figure 1 The present invention illustrates a whole-body attitude estimation method based on sparse inertial sensors, and the specific recognition steps are as follows:

[0037] Step S1: Using the finite difference method and forward kinematics algorithm, synthesize the acceleration data, orientation data, and angular velocity data required for neural network training using the AMASS dataset;

[0038] Learning-based pose estimation requires a large amount of labeled data to train the model. This study utilizes the existing AMASS dataset to synthesize inertial data (including acceleration, orientation, and angular velocity data) for neural network training as a supplementary method to increase the diversity and quantity of training data. Specifically, to simulate wearable inertial sensors in the real world, three virtual inertial sensors are placed on the surface of the SMPL model at the grid vertices corresponding to the head joint, left wrist joint, and right wrist joint, respectively, to capture the sequential grid position of each virtual sensor. Then, the finite difference method is used to calculate the acceleration data at the same grid position in consecutive frames.

[0039]

[0040] Where, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in frame t, i = 1, 2, 3, Δt is the time interval between two consecutive frames, and n is the smoothing factor. Setting n = 4 compared to adjacent frames (n = 1) yields acceleration data with the greatest advantage in smoothness and realism. For synthesized orientation data, the orientation data of the three virtual inertial sensors are calculated from the real joint pose using a forward kinematics algorithm according to the SMPL model.

[0041] R i (t)=FK(p(t))

[0042] Where FK(·) is the forward kinematics algorithm, R i p(t) represents the orientation data of the i-th virtual inertial sensor, and p(t) represents the actual attitude data of all joints in the body. For the synthesized angular velocity data, the aforementioned orientation data is used to calculate:

[0043] W i (t)=R i (t-1) -1 R i (t)

[0044] Among them, W i (t) and R i (t) represents the angular velocity and orientation data of the i-th virtual inertial sensor in frame t, respectively. Since the continuous 6D three-dimensional rotation representation is suitable for training and learning neural networks, the last column of the rotation matrix representation is discarded to obtain the 6D three-dimensional rotation representation of the orientation and angular velocity data.

[0045] Step S2: Align the raw acceleration and orientation data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint using a rotation transformation matrix to transform them into a usable unified coordinate system.

[0046] Three inertial sensors were used to capture motion data of the head joint, left wrist joint, and right wrist joint, respectively. Figure 2 As shown, the inertial sensors used provide coordinates F in each sensor coordinate system. S The raw acceleration data and the global inertial coordinate system F relative to each sensor. I The raw orientation data. To ensure that the data acquired by the inertial sensor is consistent with the motion in the real world, the raw inertial data (including raw acceleration data and raw orientation data) needs to be transformed into a usable unified coordinate system, which also involves the SMPL coordinate system F. M and skeletal coordinate system F B .

[0047] First, a transformation matrix needs to be found to establish the global inertial coordinate system F. I and SMPL coordinate system F M The relationship between them is represented by R. MI :F I →F M Place any inertial sensor on the head joint, ensuring the sensor axis is aligned with the SMPL body frame, thus establishing the sensor coordinate system F. S and SMPL coordinate system F M It is identical. Since the orientation data from an inertial sensor is represented as R... IS :F S →F I The orientation reading R of the inertial sensor at the head position Head The inverse matrix is ​​considered to be from F I To F M The transformation matrix R MI , represented as:

[0048] R MS =R MI R IS =inv(R Head )R IS

[0049] Where inv(·) represents the inverse operation, and the resulting R MS :F S →F M The mapping relationship from the sensor coordinate system to the SMPL coordinate system is described.

[0050] Next, the remaining two inertial sensors were worn on the left and right wrist joints in arbitrary orientations, while maintaining a uniform T-shaped posture. Acceleration and orientation data for each inertial sensor were collected for approximately 3 seconds to obtain data relative to the sensor coordinate system F. S initial acceleration and relative to the global inertial coordinate system F I initial direction Since the orientation of the inertial sensor is not the true orientation of the skeleton, it is necessary to find the SMPL coordinate system F. M To the skeletal coordinate system F B The mapping. Assume an angular deviation R between the sensor and the skeleton. BS It is constant, so we get:

[0051]

[0052] in, This represents the skeleton relative to the global inertial coordinate system F. I The initial orientation. Because the initial pose is uniform, the orientation of the entire skeleton can be calculated using the SMPL model; therefore, the orientation of the skeleton in the SMPL coordinate system is... This is known. For any pose, the absolute orientation of the bones in different coordinate systems is equivalent, resulting in:

[0053]

[0054] Among them, F M =F I R MI Therefore, the virtual skeleton orientation can be obtained, which is the orientation data of the inertial sensor in the skeleton coordinate system:

[0055] R BM =R BS R SM

[0056] For the raw acceleration data, it is first transformed into the global inertial coordinate system F based on the raw orientation data from the inertial sensor. I ,Right now Then acceleration data After subtracting gravity, transform to the SMPL coordinate system F. M The acceleration data from the inertial sensor in the SMPL coordinate system F M and skeletal coordinate system F B The sensors are securely fastened to the body with straps, ensuring they are equal in size. Through these steps, the raw inertial data is successfully transferred from their respective coordinate systems (sensor coordinate system and global inertial coordinate system) to the skeletal coordinate system, such as... Figure 3As shown, the acceleration data is scaled by 30 times to match the scale of the orientation data.

[0057] Step S3: By designing an encoder that combines a Transformer Encoder and a bidirectional long short-term memory network, the depth features of inertial data (including acceleration data, orientation data, and angular velocity data) are extracted. A decoder based on a multilayer perceptron is designed to calculate the position of the joints relative to the lumbar joint and the rotation relative to the parent joint. The human kinematics model is combined and the rotation of different body parts relative to the parent joint is estimated in multiple stages to achieve whole-body posture estimation.

[0058] Pose estimation is typically performed through the hierarchical relationships of the kinetic chain, using inertial input to estimate starting from the head joint, left wrist joint, and right wrist joint, which can quickly obtain pose information for local parts. This pose information is then combined with a human kinematic model for joint optimization, estimating the upper body and whole-body pose gradually in a multi-stage manner. This joint optimization can improve the accuracy and consistency of overall pose estimation, such as... Figure 4 As shown.

[0059] The first stage involves estimating the positions of the head joint, left wrist joint, and right wrist joint relative to the lumbar joint from inertial data. Then, based on the obtained relative joint positions, the rotations of the head joint, left elbow joint, and right elbow joint relative to their parent joints, as well as the global rotation of the lumbar joint, are estimated. The specific implementation is as follows:

[0060] For estimating the relative positions of the joints, the input is the connection vector of inertial data from three sensors worn on the user's head joint, left wrist joint, and right wrist joint. A position estimation network is designed by making full use of past and future time information, represented as b = S BN (X), in which an encoder combining a Transformer Encoder and a bidirectional long short-term memory network is designed to learn from inertial data. Based on the global and local temporal dependency depth features, a multilayer perceptron-based decoder is designed to estimate the positions of the three joints relative to the lumbar joint. By comparing the predicted and actual joint relative positions, the joint relative position loss for the head joint, left wrist joint, and right wrist joint is obtained, which is expressed as: in, For the predicted relative joint positions, b GT This represents the actual relative positions of the joints.

[0061] The location estimation network S BN The (·) structure contains an encoder and a decoder: the encoder consists of a linear layer, a Transformer Encoder, and a bidirectional long short-term memory network, and the decoder consists of a linear layer.

[0062] For joint posture estimation, the inertial data and the joint's position relative to the lumbar joint are connected as follows: As the first attitude estimation network S PN1 The input of (·), S PN1 (·) Predict 6D 3D rotation representation of the head joint, left elbow joint, and right elbow joint relative to the parent joint. and global rotation of the lumbar joint By connecting the pose vectors of the lower back joint, head joint, left elbow joint, and right elbow joint, we can quickly obtain the pose information of local areas. The loss function for joint rotation is expressed as: The weight λ is set to 0.1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. For the predicted rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint, This represents the actual rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint. The first pose estimation network S... PN1 (·) and the location estimation network S BN The structure of (·) is the same.

[0063] In the second stage, the pose of the upper body joints is estimated using inertial data and the pose vectors of the waist joint, head joint, left elbow joint, and right elbow joint. The inertial data and the first pose estimation network S are then used. PN1 The outputs of (·) are connected in series. X (2) This represents the input for the second stage. The second pose estimation network S... PN2 (·) Outputs the rotation of the upper body joints relative to the parent joint. and global rotation of the lumbar joint The loss function for the concatenated upper body joint pose vectors is as follows: The weight λ is set to 0.1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. For the predicted rotation of the upper body joints relative to the parent joint. This represents the actual rotation of the upper body joints relative to the parent joint. The second pose estimation network S PN2 (·) and the location estimation network S BN The structure of (·) is the same.

[0064] In the third stage, inertial measurements and upper body joint pose vectors are used to estimate the pose of all joints in the body. The inertial data and the second pose estimation network S are then used. PN2 The outputs of (·) are connected in series. X(3) This represents the input for the third stage. Similarly, the third-pose regression network S... PN3 The output of (·) is the rotation of all joints in the body relative to the parent joint. and global rotation of the lumbar joint The loss function for the concatenated whole-body joint pose vectors is calculated as follows: The weight λ is set to 0.1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. This refers to the predicted rotation of all joints relative to their parent joints. This represents the actual rotation of all joints relative to their parent joints. The third pose estimation network S... PN3 (·) and the location estimation network S BN The structure of (·) is the same.

[0065] For a given inertial data input, linear embedding is applied to expand the input dimension to 256, which is then fed into an encoder combining a Transformer Encoder and a bidirectional Long Short-Term Memory (LSTM) network to learn motion information. First, a Transformer Encoder is used to extract features, with a head parameter set to 8 and a self-attention layer count of 3. Then, the output generated by the Transformer Encoder is assigned as the unit state input to the first frame of two bidirectional LSTM networks with a width of 256, resulting in a 512-dimensional feature vector. The decoder shares the high-dimensional pose features from the encoder and maps them to the corresponding output dimensions through a linear operation with a ReLU activation function, ultimately outputting SMPL pose parameters in a 6D three-dimensional rotation representation. During training, forward kinematics is used to calculate the global position of the corresponding joints based on the output SMPL pose parameters, and the joint pose is further minimized based on the error between the actual and predicted joint positions.

[0066] Experimental data

[0067] We quantitatively evaluated the proposed method using the following evaluation metrics: (1) Mean Joint Rotation Error (MJRE): the average angular error of all joints between the estimated global rotation and the actual global rotation, in degrees; (2) MJRE-Lumbar Joint: the angular error between the estimated global rotation of the lumbar joint and the actual global rotation of the lumbar joint, in degrees; (3) Mean Joint Position Error (MJPE): the average Euclidean distance error between the estimated position of the whole body joints relative to the lumbar joint and the actual position of the whole body joints relative to the lumbar joint, in centimeters; (4) MJPE-Wrist: the average Euclidean distance error between the estimated position of the left and right wrist joints relative to the lumbar joint and the actual position of the left and right wrist joints relative to the lumbar joint, in centimeters; (5) Mesh Error (ME): the average Euclidean distance error of all mesh vertices in the SMPL model relative to the lumbar joint mesh vertices, in centimeters.

[0068] We selected three baselines most similar to our method from state-of-the-art methods for estimating whole-body posture from inertial data: AGLOL, IMUPoser, and TransPose. Table 1 shows the online performance comparison between the proposed method and the three baselines on the AMASS dataset. As can be seen from the table, the proposed method performs excellently on all five metrics: MJRE, MJRE-lumbar joint, MJPE, MJPE-wrist, and ME, achieving 12.42 degrees, 13.76 degrees, 7.52 cm, 8.22 cm, and 7.45 cm respectively, significantly outperforming other methods. Compared to the suboptimal method IMUPoser, our model reduces the MJRE-lumbar joint, MJPE, MJPE-wrist, and ME metrics by 1.9 degrees, 1.07 cm, 4.16 cm, and 1.77 cm respectively. Furthermore, it achieves 12.42 degrees on the MJPE-wrist metric, lower than the suboptimal AGLOL. Experimental results demonstrate the stability and superiority of the proposed method, which can effectively capture subtle differences in challenging movements, especially wrist movements and lumbar rotation.

[0069] Table 1: Online performance comparison of the proposed method with three baselines on the AMASS dataset.

[0070]

[0071] This invention utilizes three Noitom inertial sensors worn on the user's head, left wrist, and right wrist to create a series of real-time animated demonstrations, verifying the effectiveness and feasibility of the method in full-body posture estimation. In these demonstrations, the user wears the Noitom inertial sensors and performs various actions, such as... Figure 5As shown, three inertial sensors are connected to a real-time data processing system on a computer via Bluetooth, and the animation is rendered in real-time within Unity. This method can accurately estimate the motion of all joints and reflect it in real time on the virtual character, displaying highly realistic and smooth animation in the virtual environment.

Claims

1. A method for whole-body attitude estimation based on sparse inertial sensors, characterized in that, Includes the following steps: Step S1: Synthesize the acceleration data, orientation data, and angular velocity data required for neural network training using the finite difference method and forward kinematics algorithm; Step S2: Align the raw acceleration and orientation data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint using a rotation transformation matrix to transform them into a usable unified coordinate system. Step S3: Extract depth features from inertial motion data samples using an encoder combining a TransformerEncoder and a bidirectional long short-term memory network. Design a decoder based on a multilayer perceptron to calculate joint positions and relative rotations. Combine this with a human kinematics model and estimate the relative rotations of joints at different locations in multiple stages to achieve whole-body posture estimation. Specific steps include: S3.1: The depth features of the inertial data are extracted by designing an encoder that combines a Transformer Encoder and a bidirectional long short-term memory network. A decoder based on a multilayer perceptron is designed to calculate the position of the joints relative to the lumbar joint and the rotation relative to the parent joint. The rotation of the joints of different body parts relative to the parent joint is estimated in multiple stages by combining the human kinematic model. The inertial data includes acceleration data, orientation data, and angular velocity data. S3.2: Combining a human kinematics model, the rotation of local key joints, upper body joints, and whole-body joints relative to the parent joint, as well as the global rotation of the lumbar joint, are estimated sequentially in multiple stages. Specifically: The first stage realizes the estimation from the inertial measurement value X to the local key joint pose p1 and the global orientation p. (1) The mapping process involves the position b of a local key joint as an intermediate step; the second stage estimates the upper body joint pose p2 and global orientation p from the inertial measurement value X and the local key joint pose p1 output from the first stage. (2) The third stage involves estimating the whole-body joint pose p3 and global orientation p3 from the inertial measurement value X and the upper-body joint pose p2 output from the second stage. (3) .

2. The whole-body attitude estimation method based on sparse inertial sensors according to claim 1, characterized in that, Step S1 includes the following specific steps: S1.1: Place virtual inertial sensors on the mesh vertices corresponding to the head joint and the left and right wrist joints on the surface of the SMPL model, and use finite difference to calculate synthetic acceleration data at the same mesh position in adjacent time frames of the virtual inertial sensors: Where, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in frame t, i = 1, 2, 3, Δt is the time interval between two consecutive frames, and n is the smoothing factor; S1.2: Using the forward kinematics algorithm, the synthetic orientation data of the three virtual inertial sensors is calculated from the real joint posture according to the SMPL model; R i (t)=FK(p(t)) Where FK(·) is the forward kinematics algorithm, R i p(t) is the orientation data of the i-th virtual inertial sensor, and p(t) is the actual posture data of all joints in the body; S1.3: Calculate the composite angular velocity data using the composite direction data: W i (t)=R i (t-1) -1 R i (t) Among them, W i (t) and R i (t) represents the synthesized direction data and synthesized angular velocity data of the i-th virtual inertial sensor in frame t, respectively.

3. The whole-body attitude estimation method based on sparse inertial sensors according to claim 1, characterized in that, Step S2 includes the following specific steps: S2.1: A transformation matrix needs to be found to establish the global inertial coordinate system F. I and SMPL coordinate system F M The relationship between them is represented by R. MI :F I →F M Select any inertial sensor to wear on the head joint, ensuring that the sensor axis is aligned with the SMPL body frame, so that the sensor coordinate system F S and SMPL coordinate system F M It is identical; since the orientation data of the inertial sensor is represented by R. IS :F S →F I The orientation reading R of the inertial sensor at the head position Head The inverse matrix is ​​considered to be from F I To F M The transformation matrix R MI , is represented as: R MS =R MI R IS =inv(R Head )R IS Where inv(·) represents the inverse operation, and the resulting R MS :F S →F M The mapping relationship from the sensor coordinate system to the SMPL coordinate system is described; S2.2: Wear the remaining two inertial sensors in any direction while maintaining a uniform T-shaped posture to obtain the coordinates relative to the sensor coordinate system F. S initial acceleration and relative to the global inertial coordinate system F I initial direction The remaining two inertial sensors are worn on the left and right wrist joints in arbitrary orientations, while the body maintains a uniform T-shaped posture, in order to obtain data relative to the sensor coordinate system F. S initial acceleration and relative to the global inertial coordinate system F I initial direction Let R be the angular deviation between the sensor and the skeleton. BS If it is constant, then the skeleton's position relative to the global inertial coordinate system F is obtained. I initial direction in, This represents the skeleton relative to the global inertial coordinate system F. I The initial direction; S2.3: Since the initial pose is uniform, the orientation of the entire skeleton can be calculated using the SMPL model. Therefore, the orientation of the skeleton in the SMPL coordinate system is... It is known that for any pose, the absolute orientation of the bones is equivalent in different coordinate systems, thus: Among them, F M =F I R MI Therefore, the virtual skeleton orientation can be obtained, which is the orientation data of the inertial sensor in the skeleton coordinate system: R BM =R BS R SM Among them, R BM This represents the orientation data of the inertial sensor in the skeletal coordinate system; S2.4: For the raw acceleration data, first transform it to the global inertial coordinate system F based on the raw orientation data from the inertial sensor. I ,Right now Then acceleration data After subtracting gravity, transform to the SMPL coordinate system F. M The acceleration data from the inertial sensor in the SMPL coordinate system F M and skeletal coordinate system F B Since they are equal, they are simultaneously transformed to the skeletal coordinate system F. B .

4. The whole-body attitude estimation method based on sparse inertial sensors according to claim 3, characterized in that, The first phase will be implemented as follows: First, the system estimates the positions of the head joint, left wrist joint, and right wrist joint relative to the waist joint from inertial data. Then, based on the obtained relative joint positions, it estimates the rotations of the head joint, left elbow joint, and right elbow joint relative to the parent joint, as well as the global rotation of the waist joint. The specific implementation is as follows: For estimating the relative positions of the joints, the input is the connection vector of inertial data from three sensors worn on the user's head joint, left wrist joint, and right wrist joint. A position estimation network is designed by making full use of past and future time information, represented as b = S BN (X), in which an encoder combining a Transformer Encoder and a bidirectional long short-term memory network is designed to learn habitual data. Based on the global and local temporal dependency deep features, a multilayer perceptron-based decoder is designed to estimate the positions of the three joints relative to the lumbar joint. By comparing the predicted and actual joint relative positions, the joint relative position loss for the head joint, left wrist joint, and right wrist joint is obtained, which is expressed as: in, For the predicted relative joint positions, b GT This represents the actual relative positions of the joints; For joint posture estimation, the inertial data and the joint's position relative to the lumbar joint are connected as follows: As the first attitude estimation network S PN1 The input of (·), S PN1 (·) Predict 6D 3D rotation representation of the head joint, left elbow joint, and right elbow joint relative to the parent joint. and global rotation of the lumbar joint The concatenated head joint, left elbow joint, and right elbow joint pose vectors quickly provide pose information for local areas; the loss function for joint rotation is expressed as: The weight λ is set to 0.

1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. For the predicted rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint. This represents the actual rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint.

5. The whole-body attitude estimation method based on sparse inertial sensors according to claim 4, characterized in that, The location estimation network S BN The (·) structure contains an encoder and a decoder: the encoder consists of a linear layer, a Transformer Encoder, and a bidirectional long short-term memory network, and the decoder consists of a linear layer.

6. A method for whole-body attitude estimation based on sparse inertial sensors according to claim 4 or 5, characterized in that, The second phase will be implemented as follows: The pose of the upper body joints is estimated using inertial data and the pose vectors of the head joint, waist joint, left elbow joint, and right elbow joint; the inertial data and the first pose estimation network S are then used. PN1 The outputs of (·) are connected in series. X (2) This represents the input for the second stage; the second pose estimation network S PN2 (·) Outputs the rotation of the upper body joints relative to the parent joint. and global rotation of the lumbar joint The loss function for the concatenated upper body joint pose vectors is as follows: The weight λ is set to 0.

1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. For the predicted rotation of the upper body joints relative to the parent joint. This represents the actual rotation of the upper body joints relative to the parent joint.

7. The whole-body attitude estimation method based on sparse inertial sensors according to claim 6, characterized in that, The third phase will be implemented as follows: The attitude of all joints is estimated using inertial measurements and upper body joint pose vectors; the inertial data and the second attitude estimation network S are then used. PN2 The outputs of (·) are connected in series. X (3) This represents the input for the third stage; the third-pose regression network S PN3 The output of (·) is the rotation of all joints in the body relative to the parent joint. and global rotation of the lumbar joint The loss function for the concatenated whole-body joint pose vectors is calculated as follows: The weight λ is set to 0.

1. For predicting global rotation of the lumbar joint, This represents the actual global rotation of the lumbar joint. This refers to the predicted rotation of all joints relative to their parent joints. This represents the rotation of the actual joints of the whole body relative to the parent joint.

8. The whole-body attitude estimation method based on sparse inertial sensors according to claim 7, characterized in that, Second pose estimation network S PN2 (·) and the location estimation network S BN The structure of (·) is the same, and the third pose estimation network S PN3 (·) and the location estimation network S BN The structure of (·) is the same.

Images