Whole body attitude estimation method based on sparse inertial sensor

Abstract

The invention discloses a whole body attitude estimation method based on a sparse inertial sensor. The method comprises the following steps: 1, synthesizing acceleration data, direction data and angular velocity data through a finite difference method and a forward kinematics algorithm; 2, performing alignment processing on original acceleration data and original direction data acquired by three inertial sensors worn on a head joint, a left wrist joint and a right wrist joint of the user by rotating the transformation matrix; and 3, extracting depth features of an inertial motion data sample through an encoder combining Transform Encoder and a bidirectional long-short time memory network, designing a decoder based on a multilayer perceptron to calculate positions and relative rotation of joints, and sequentially estimating the relative rotation of the joints of different parts in a plurality of stages in combination with a human body kinematics model so as to realize whole body posture estimation. The method can effectively overcome the defect that sparse inertia input is used for whole body motion reconstruction in an existing attitude estimation method, and the robustness and prediction precision of the system are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion capture, in particular to posture estimation, and specifically relates to a whole body posture estimation method based on sparse inertial sensors. Background Art

[0002] The fusion innovation of virtual reality and motion capture can provide users with real-time feedback and high interactivity, ensuring accurate mapping and matching of real movements and virtual scenes, allowing users to participate more directly in the virtual environment, thus achieving a seamless integration of reality and virtuality.

[0003] Vision-based motion capture applications are mature and occupy most of the market share. Motion capture is achieved by monitoring and tracking target feature points from different angles using multiple RGB cameras in real-time image streams. This solution usually has high real-time and responsiveness, but is easily affected by external environments such as lighting conditions and occlusions. The capture accuracy of optical systems based on markers is widely recognized. Its principle is to detect markers attached to human joints in real time through motion capture cameras, but it requires large installations and complex calibration procedures. With the rapid advancement of micro-electromechanical system technology, wearable sensors provide a novel solution for motion capture. Motion capture systems based on wearable sensors are not limited by space and environment, avoiding the problem of monitoring area and environmental occlusion. The miniaturization of sensors also increases the comfort of users' daily wear and is more suitable for applications in mobile scenarios. However, commercial inertial motion capture systems such as Xsens obtain joint acceleration, angular velocity and magnetic field strength from 17 or more inertial sensors. Especially when used for a long time, too many sensors will limit the diversity of movements and user comfort, and the management cost of multiple devices and the complexity of calculations will increase. Summary of the invention

[0004] In order to overcome the problem that the prior art only uses sparse inertial inputs from the head joint and the left and right wrist joints and is not applicable to all whole body posture estimations, the present invention provides a whole body posture estimation method based on sparse inertial sensors.

[0005] To achieve the above object, the present invention provides the following technical solution: a whole body posture estimation method based on sparse inertial sensors, the specific identification steps are as follows:

[0006] Step S1: synthesizing acceleration data, direction data and angular velocity data required for neural network training through finite difference method and forward kinematics algorithm;

[0007] Step S2: aligning the original acceleration data and original direction data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint through a rotation transformation matrix to transform them into an available unified coordinate system;

[0008] Step S3: By combining the Transformer Encoder and the bidirectional long short-term memory network encoder to extract the deep features of the inertial motion data samples, a decoder based on a multi-layer perceptron is designed to calculate the position and relative rotation of the joints. The human kinematic model is combined and the relative rotation of the joints of different parts is estimated in multiple stages to achieve whole-body posture estimation.

[0009] Furthermore, step S1 specifically includes the following steps:

[0010] S1.1: Place virtual inertial sensors at the mesh vertices corresponding to the head joint and left and right wrist joints on the surface of the SMPL model, and use finite differences to calculate synthetic acceleration data for the same mesh position of the virtual inertial sensors in adjacent time frames:

[0011]

[0012] Among them, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in the t-th frame, i = 1, 2, 3, Δt is the time interval between two consecutive frames, and n is the smoothing factor;

[0013] S1.2: Using the forward kinematics algorithm, the synthetic orientation data of the three virtual inertial sensors are calculated from the real joint posture according to the SMPL model;

[0014] R i (t) = FK(p(t))

[0015] Among them, FK(·) is the forward kinematics algorithm, R i (t) is the direction data of the i-th virtual inertial sensor, and p(t) is the real posture data of the whole body joints;

[0016] S1.3: Calculate the synthetic angular velocity data using the synthetic direction data:

[0017] W i (t) = R i (t-1) -1 R i (t)

[0018] Among them, W i (t) and R i(t) represent the synthetic direction data and synthetic angular velocity data of the i-th virtual inertial sensor in the t-th frame respectively.

[0019] Furthermore, step S2 specifically includes the following steps:

[0020] S2.1: We need to find a transformation matrix to establish the global inertial coordinate system F I and SMPL coordinate system F M The relationship between them is represented by R MI :F I →F M ; Choose any inertial sensor to wear on the head joint and make sure the sensor axis is consistent with the SMPL body frame, so that the sensor coordinate system F S and SMPL coordinate system F M is the same; since the direction data of the inertial sensor is expressed as R IS :F S →F I , the orientation reading R of the inertial sensor at the head position Head The inverse matrix of F is considered to be I to F M The transformation matrix R MI ;

[0021] S2.2: Wear the remaining two inertial sensors in any orientation, keeping the body in a uniform T-shaped posture, in order to obtain the coordinates relative to the sensor coordinate system F S The initial acceleration and relative to the global inertial coordinate system F I The initial direction

[0022] The remaining two inertial sensors are worn on the left and right wrist joints in any direction, and the body maintains a uniform T-shaped posture to obtain the coordinate system F relative to the sensor. S The initial acceleration and relative to the global inertial coordinate system F I The initial direction Assume the angle deviation between the sensor and the skeleton is R BS is constant, then we get the bone relative to the global inertial coordinate system F I The initial direction

[0023] S2.3: Since the initial pose is uniform, the orientation of the whole body skeleton can be calculated by the SMPL model, so the orientation of the skeleton in the SMPL coordinate system It is known that for any posture, the absolute orientation of the bones in different coordinate systems is equivalent, and we get Because F M=F I R MI , so the virtual skeleton direction can be obtained, that is, the direction data of the inertial sensor in the skeleton coordinate system.

[0024] S2.4: For the raw acceleration data, first convert it to the global inertial coordinate system F according to the raw direction data of the inertial sensor I ,Right now Then the acceleration data After subtracting gravity, transform to SMPL coordinate system F M , the acceleration data of the inertial sensor is in the SMPL coordinate system F M and the bone coordinate system F B are equal, so they are transformed to the bone coordinate system F at the same time B .

[0025] Furthermore, the specific steps of step S3 include:

[0026] S3.1: Extract the deep features of inertial data (including acceleration data, orientation data, and angular velocity data) by designing an encoder combining Transformer Encoder and bidirectional long short-term memory network, design a decoder based on multi-layer perceptron to calculate the position of the joint relative to the waist joint and the rotation relative to the parent joint, combine the human kinematic model and estimate the rotation of the joints of different body parts relative to the parent joint in multiple stages; the inertial data includes acceleration data, orientation data, and angular velocity data;

[0027] S3.2: Combined with the human kinematics model, the rotation of local key joints, upper body joints and whole body joints relative to the parent joints and the global rotation of the waist joint are estimated in multiple stages, where: the first stage realizes the inertial measurement value X to the local key joint posture p1 and the global direction p (1) The second stage estimates the upper body joint posture p2 and the global direction p from the inertial measurement value X and the local key joint posture p1 output in the first stage. (2) The third stage estimates the whole body joint posture p3 and the global direction p from the inertial measurement value X and the upper body joint posture p2 output in the second stage. (3) .

[0028] Technical effects and advantages of the present invention:

[0029] The present invention uses the acceleration and direction measurements provided by three inertial sensors from the user's head joint and the left and right head wrist joints as input, and designs a whole-body posture estimation method based on sparse inertial sensors. First, the advantages of local inertial data are used to estimate the posture of local key joints, and then gradually extend to the core parts of the upper body such as the pelvis and torso, and finally achieve accurate reconstruction of whole-body motion. At the posture estimation level, a decoder combining TransformerEncoder and bidirectional long short-term memory network is specially designed to improve the understanding of inertial motion information, and a decoder based on a multi-layer perceptron is used to share the rich and comprehensive high-dimensional posture features from the encoder, and the sparsely tracked acceleration and direction measurements are fully mapped to the whole-body joints of the SMPL model. A sliding window-based test strategy that is conducive to real-time architecture deployment and past and future motion information are used to predict the current posture, and an animation demonstration effect with acceptable delay is generated in real time. The method proposed in the present invention can effectively overcome the defects of using sparse inertial input for whole-body motion reconstruction in existing posture estimation methods, and effectively improve the robustness and prediction accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart for implementing the present invention.

[0031] Figure 2 FIG. 1 is a schematic diagram of wearing an inertial sensor applicable to the present invention.

[0032] Figure 3 Align the schematic for inertial sensor calibration.

[0033] Figure 4 Flowchart for full body pose estimation.

[0034] Figure 5 Animated demonstration of real-time pose estimation. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The present invention provides Figure 1 A whole body posture estimation method based on sparse inertial sensors is shown, and the specific recognition steps are as follows:

[0037] Step S1: Using the finite difference method and forward kinematics algorithm, the AMASS data set is used to synthesize the acceleration data, direction data and angular velocity data required for neural network training;

[0038] Learning-based posture estimation requires a large amount of labeled data to train the model. The existing AMASS dataset is used to synthesize inertial data (including acceleration data, orientation data, and angular velocity data) for neural network training as a supplementary means to increase the diversity and quantity of training data. Specifically, in order to simulate the inertial sensors worn by users in the real world, three virtual inertial sensors are placed on the mesh vertices corresponding to the head joint, left wrist joint, and right wrist joint on the surface of the SMPL model to capture the sequential grid position of each virtual sensor, and then the finite difference method is used to calculate the same grid position of consecutive frames to synthesize acceleration data:

[0039]

[0040] Among them, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in the t-th frame, i = 1, 2, 3, Δt is the time interval between two consecutive frames, n is the smoothing factor, and setting n = 4 can obtain acceleration data with the greatest advantages in smoothness and authenticity compared to adjacent frames (n = 1). For the synthetic direction data, the forward kinematics algorithm is used to calculate the direction data of the three virtual inertial sensors from the real joint posture according to the SMPL model.

[0041] R i (t) = FK(p(t))

[0042] Among them, FK(·) is the forward kinematics algorithm, R i (t) is the direction data of the i-th virtual inertial sensor, and p(t) is the real posture data of the whole body joints. For the synthetic angular velocity data, the direction data is used to calculate:

[0043] W i (t) = R i (t-1) -1 R i (t)

[0044] Among them, W i (t) and R i (t) represents the angular velocity data and direction data of the i-th virtual inertial sensor in the t-th frame. Since the continuous 6D three-dimensional rotation representation is suitable for the training and learning of the neural network, the last column of the rotation matrix representation is discarded to obtain the 6D three-dimensional rotation representation of the direction data and angular velocity data.

[0045] Step S2: aligning the original acceleration data and original direction data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint through a rotation transformation matrix to transform them into an available unified coordinate system;

[0046] Use three inertial sensors to capture the motion data of the head joint, left wrist joint, and right wrist joint respectively. Figure 2 As shown, the inertial sensors used provide a coordinate system F in each sensor. S The raw acceleration data and the global inertial coordinate system F located at each sensor relative to I To ensure that the data obtained by the inertial sensor is consistent with the movement in the real world, it is necessary to transform the raw inertial data (including raw acceleration data and raw direction data) into an available unified coordinate system, which also involves the SMPL coordinate system F M and the bone coordinate system F B .

[0047] First, we need to find a transformation matrix to establish the global inertial coordinate system F I and SMPL coordinate system F M The relationship between them is represented by R MI :F I →F M Place any inertial sensor on the head joint and ensure that the sensor axis is aligned with the SMPL body frame, so that the sensor coordinate system F S and SMPL coordinate system F M is the same. Since the direction data of the inertial sensor is expressed as R IS :F S →F I , the orientation reading R of the inertial sensor at the head position Head The inverse matrix of F is considered to be I to F M The transformation matrix R MI , expressed as:

[0048] R MS =R MI R IS =inv(R Head )R IS

[0049] Among them, inv(·) represents the inverse operation, and the obtained R MS :F S →F M Describes the mapping relationship from the sensor coordinate system to the SMPL coordinate system.

[0050] Next, the remaining two inertial sensors are worn on the left and right wrist joints in any direction, the body maintains a uniform T-shaped posture, and the acceleration and direction data of each inertial sensor are collected within about 3 seconds to obtain the relative to the sensor coordinate system F S The initial acceleration and relative to the global inertial coordinate system F I The initial direction Since the direction of the inertial sensor is not the true direction of the skeleton, it is necessary to find the SMPL coordinate system F M To the bone coordinate system F B Assume that the angular deviation between the sensor and the skeleton is R BS is constant, we get:

[0051]

[0052] in, Represents the bone relative to the global inertial coordinate system F I Because the initial posture is unified, the whole body skeleton direction can be calculated by the SMPL model, so the direction of the skeleton in the SMPL coordinate system is is known. For any posture, the absolute orientation of the bones in different coordinate systems is equivalent, and we get:

[0053]

[0054] Among them, F M =F I R MI , so we can get the virtual skeleton direction, that is, the direction data of the inertial sensor in the skeleton coordinate system:

[0055] R BM =R BS R SM

[0056] For the raw acceleration data, first convert it to the global inertial coordinate system F according to the raw direction data of the inertial sensor I ,Right now Then the acceleration data After subtracting gravity, transform to SMPL coordinate system F M The acceleration data of the inertial sensor is in the SMPL coordinate system F M and the bone coordinate system F B After the above steps, the raw inertial data are successfully transferred from their own coordinate systems (sensor coordinate system and global inertial coordinate system) to the skeleton coordinate system, such as Figure 3As shown, the acceleration data is scaled 30 times to match the scale of the orientation data.

[0057] Step S3: By designing an encoder that combines the Transformer Encoder and the bidirectional long short-term memory network to extract the deep features of the inertial data (including acceleration data, orientation data, and angular velocity data), a decoder based on a multi-layer perceptron is designed to calculate the position of the joint relative to the waist joint and the rotation relative to the parent joint. The human kinematic model is combined and the rotation of the joints of different body parts relative to the parent joint is estimated in sequence in multiple stages to achieve whole-body posture estimation.

[0058] Posture estimation is usually performed through the hierarchical relationship of the kinematic chain. Using inertial input, it starts with the head joint, left wrist joint, and right wrist joint to quickly obtain the posture information of local parts. Then, this posture information is combined with the human kinematic model for joint optimization, and the posture of the upper body and the whole body is gradually estimated in a multi-stage manner. This joint optimization can improve the accuracy and consistency of the overall posture estimation, such as Figure 4 shown.

[0059] In the first stage, we first estimate the position of the head joint, left wrist joint, and right wrist joint relative to the waist joint from the inertial data; then we estimate the rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint and the global rotation of the waist joint based on the obtained joint relative positions. The specific implementation is as follows:

[0060] For the estimation of the relative position of joints, the input is the connection vector of the inertial data of three sensors worn on the user's head joint, left wrist joint, and right wrist joint. A position estimation network is designed by making full use of past and future time information, expressed as b = S BN (X), where an encoder combining a Transformer Encoder and a bidirectional long short-term memory network is designed to learn from inertial data Based on the global and local temporal dependency deep features in the training data, a decoder based on a multi-layer perceptron is designed to estimate the positions of the three joints relative to the waist joint. Through the predicted relative positions of the joints and the actual relative positions of the joints, the joint relative position loss of the head joint, left wrist joint, and right wrist joint is obtained, which is expressed as: in, is the predicted relative position of the joint, b GT is the real relative position of the joint.

[0061] The position estimation network S BN (·) The structure consists of an encoder and a decoder: the encoder consists of a linear layer, a Transformer Encoder, and a bidirectional long short-term memory network, and the decoder consists of a linear layer.

[0062] For the estimation of joint posture, the inertial data and the position of the joint relative to the waist joint are connected as As the first pose estimation network S PN1 (·) input, S PN1 (·) Predict the rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint in the 6D three-dimensional rotation representation and the global rotation of the waist joint Connect the posture vectors of the lower back joint, head joint, left elbow joint, and right elbow joint, so that the posture information of the local parts can be quickly obtained. The loss function of the joint rotation is expressed as: Among them, the weight λ is set to 0.1, is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint, is the rotation of the real head joint, left elbow joint, and right elbow joint relative to the parent joint. The first posture estimation network S PN1 (·) and the position estimation network S BN The structure of (·) is the same.

[0063] In the second stage, the inertial data and the posture vectors of the waist joint, head joint, left elbow joint, and right elbow joint are used to estimate the posture of the upper body joints. PN1 The output of (·) is connected in series X (2) Represents the input of the second stage. The second posture estimation network S PN2 (·) Output the rotation of the upper body joint relative to the parent joint and the global rotation of the waist joint The loss function of the connected upper body joint posture vector is expressed as: Among them, the weight λ is set to 0.1. is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the upper body joint relative to the parent joint, is the real rotation of the upper body joint relative to the parent joint. The second posture estimation network S PN2 (·) and the position estimation network S BN The structure of (·) is the same.

[0064] In the third stage, the inertial measurements and upper body joint pose vectors are used to estimate the pose of the whole body joints. PN2 The output of (·) is connected in series X(3) Represents the input of the third stage. Similarly, the third posture regression network S PN3 The output of (·) is the rotation of the whole body joint relative to the parent joint and the global rotation of the waist joint After the connection, the loss function of the whole body joint posture vector is calculated as: Among them, the weight λ is set to 0.1. is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the whole body joint relative to the parent joint, is the real rotation of the whole body joint relative to the parent joint. The third posture estimation network S PN3 (·) and the position estimation network S BN The structure of (·) is the same.

[0065] For a given inertial data input, linear embedding is applied to expand the input dimension to 256 dimensions, and then it is fed to the encoder that combines Transformer Encoder and bidirectional long short-term memory network to learn motion information: first, Transformer Encoder is used to extract features, where the head parameter is set to 8 and the number of self-attention layers is set to 3; then the output generated by Transformer Encoder is assigned as the unit state input of the first frame of two bidirectional long short-term memory networks with width 256, and the output is a 512-dimensional feature vector. The decoder shares the high-dimensional posture features from the encoder and maps them to the corresponding output dimensions through linear operations with ReLU activation functions, and finally outputs the SMPL posture parameters of 6D three-dimensional rotation representation. In the training stage, forward kinematics is used to calculate the global position of the corresponding joint based on the output SMPL posture parameters, and the joint posture is further minimized based on the error between the true joint position and the predicted joint position.

[0066] Experimental data

[0067] We use the following evaluation metrics to quantitatively evaluate the proposed method: (1) Mean joint rotation error (MJRE): the average angular error between the estimated global rotation and the true global rotation of all joints, in degrees; (2) MJRE-waist joint: the angular error between the estimated global rotation of the waist joint and the true global rotation of the waist joint, in degrees; (3) Mean joint position error (MJPE): the average Euclidean distance error between the estimated positions of the whole body joints relative to the waist joint and the true positions of the whole body joints relative to the waist joint, in centimeters; (4) MJPE-wrist: the average Euclidean distance error between the estimated positions of the left wrist joint and the right wrist joint relative to the waist joint and the true positions of the left wrist joint and the right wrist joint relative to the waist joint, in centimeters; (3) Mesh error (ME): the average Euclidean distance error of all mesh vertices in the SMPL model relative to the mesh vertices of the waist joint, in centimeters.

[0068] We selected three baselines that are most similar to our method from the most advanced methods for estimating full body pose from inertial data: AGRoL, IMUPoser, and TransPose. Table 1 shows the online performance comparison between the proposed method and the three baselines on the AMASS dataset. As can be seen from the data in the table, the proposed method performs well in the five indicators of MJRE, MJRE-waist joint, MJPE, MJPE-wrist, and ME, reaching 12.42 degrees, 13.76 degrees, 7.52 cm, 8.22 cm, and 7.45 cm, respectively, which is significantly better than other methods. Compared with the second-best method IMUPoser, the model in this study reduces the MJRE-waist joint, MJPE, MJPE-wrist, and ME indicators by 1.9 degrees, 1.07 cm, 4.16 cm, and 1.77 cm, respectively. At the same time, it reaches 12.42 degrees on the MJPE-wrist indicator, which is lower than the second-best solution AGRoL. Experimental results demonstrate the stability and superiority of the proposed method in effectively capturing the nuances of challenging actions, especially wrist motion and waist joint rotation.

[0069] Table 1: Online performance comparison between the proposed method and three baselines on the AMASS dataset

[0070]

[0071] The present invention uses three Noitom inertial sensors worn on the user's head, left wrist, and right wrist to conduct a series of real-time animation demonstrations to verify the effectiveness and feasibility of the method in whole-body posture estimation. In these demonstrations, the user wears the Noitom inertial sensor and performs various actions, such as Figure 5As shown in Figure 2, three inertial sensors are connected to a real-time data processing system on a computer via Bluetooth technology and rendered in Unity for real-time animation. This method can accurately estimate the joint motion of the whole body and reflect it on the virtual character in real time, displaying highly realistic and smooth animation in the virtual environment.

Claims

1. A whole body posture estimation method based on sparse inertial sensors, characterized in that: The steps include: Step S1: synthesizing the acceleration data, direction data and angular velocity data required for neural network training through the finite difference method and forward kinematics algorithm; Step S2: aligning the original acceleration data and original direction data collected by the three inertial sensors worn on the user's head joint, left wrist joint, and right wrist joint through a rotation transformation matrix to transform them into an available unified coordinate system; Step S3: By combining the Transformer Encoder and the bidirectional long short-term memory network encoder to extract the deep features of the inertial motion data samples, a decoder based on a multi-layer perceptron is designed to calculate the position and relative rotation of the joints. The human kinematic model is combined and the relative rotation of the joints of different parts is estimated in multiple stages to achieve whole-body posture estimation.

2. A whole body posture estimation method based on sparse inertial sensors according to claim 1, characterized in that: Step S1 specifically includes: S1.1: Place virtual inertial sensors at the mesh vertices corresponding to the head joint and left and right wrist joints on the surface of the SMPL model, and use finite differences to calculate synthetic acceleration data for the same mesh position of the virtual inertial sensors in adjacent time frames: Among them, x i (t) and a i (t) represents the grid position and acceleration data of the i-th virtual inertial sensor in the t-th frame, i = 1, 2, 3, Δt is the time interval between two consecutive frames, and n is the smoothing factor; S1.2: Using the forward kinematics algorithm, the synthetic orientation data of the three virtual inertial sensors are calculated from the real joint posture according to the SMPL model; R i (t)=FK(p(t)) Among them, FK(·) is the forward kinematics algorithm, R i (t) is the direction data of the i-th virtual inertial sensor, and p(t) is the real posture data of the whole body joints; S1.3: Calculate the synthetic angular velocity data using the synthetic direction data: W i (t)=R i (t-1) -1 R i (t) Among them, W i (t) and R i (t) represent the synthetic direction data and synthetic angular velocity data of the i-th virtual inertial sensor in the t-th frame respectively.

3. The whole body posture estimation method based on sparse inertial sensors according to claim 1, characterized in that: Step S2 specifically includes: S2.1: We need to find a transformation matrix to establish the global inertial coordinate system F I and SMPL coordinate system F M The relationship between them is represented by R MI :F I →F M ; Choose any inertial sensor to wear on the head joint and make sure the sensor axis is consistent with the SMPL body frame, so that the sensor coordinate system F S and SMPL coordinate system F M is the same; since the direction data of the inertial sensor is expressed as R IS :F S →F I , the orientation reading R of the inertial sensor at the head position Head The inverse matrix of F is considered to be I to F M The transformation matrix R MI , expressed as: R MS =R MI R IS =inv(R Head )R IS Among them, inv(·) represents the inverse operation, and the obtained R MS :F S →F M Describes the mapping relationship between the sensor coordinate system and the SMPL coordinate system; S2.2: Wear the remaining two inertial sensors in any orientation, keeping the body in a uniform T-shaped posture, in order to obtain the coordinates relative to the sensor coordinate system F S The initial acceleration and relative to the global inertial coordinate system F I The initial direction The remaining two inertial sensors are worn on the left and right wrist joints in any direction, and the body maintains a uniform T-shaped posture to obtain the coordinate system F relative to the sensor. S The initial acceleration and relative to the global inertial coordinate system F I The initial direction Assume the angle deviation between the sensor and the skeleton is R BS is constant, then we get the bone relative to the global inertial coordinate system F I The initial direction in, Represents the bone relative to the global inertial coordinate system F I The initial direction of S2.3: Since the initial pose is uniform, the orientation of the whole body skeleton can be calculated by the SMPL model, so the orientation of the skeleton in the SMPL coordinate system It is known that for any posture, the absolute orientation of the bones in different coordinate systems is equivalent, and we get: Among them, F M =F I R MI , so we can get the virtual skeleton direction, that is, the direction data of the inertial sensor in the skeleton coordinate system: R BM =R BS R SM Among them, R BM Represents the orientation data of the inertial sensor in the skeleton coordinate system; S2.4: For the raw acceleration data, first convert it to the global inertial coordinate system F according to the raw direction data of the inertial sensor I ,Right now Then the acceleration data After subtracting gravity, transform to SMPL coordinate system F M , the acceleration data of the inertial sensor is in the SMPL coordinate system F M and the bone coordinate system F B are equal, so they are transformed to the bone coordinate system F at the same time B .

4. The whole body posture estimation method based on sparse inertial sensors according to claim 3 is characterized in that: The specific steps of step S3 include: S3.1: Extract the deep features of inertial data (including acceleration data, orientation data, and angular velocity data) by designing an encoder combining Transformer Encoder and bidirectional long short-term memory network, design a decoder based on multi-layer perceptron to calculate the position of the joint relative to the waist joint and the rotation relative to the parent joint, combine the human kinematic model and estimate the rotation of the joints of different body parts relative to the parent joint in multiple stages; the inertial data includes acceleration data, orientation data, and angular velocity data; S3.2: Combined with the human kinematics model, the rotation of local key joints, upper body joints and whole body joints relative to the parent joints and the global rotation of the waist joint are estimated in multiple stages, where: the first stage realizes the inertial measurement value X to the local key joint posture p1 and the global direction p (1) The second stage estimates the upper body joint posture p2 and the global direction p from the inertial measurement value X and the local key joint posture p1 output in the first stage. (2) The third stage estimates the whole body joint posture p3 and the global direction p from the inertial measurement value X and the upper body joint posture p2 output in the second stage. (3) .

5. A whole body posture estimation method based on sparse inertial sensors according to claim 4, characterized in that: The first stage is implemented as follows: First, estimate the position of the head joint, left wrist joint, and right wrist joint relative to the waist joint from inertial data; then estimate the rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint and the global rotation of the waist joint based on the obtained joint relative positions. The specific implementation is as follows: For the estimation of the relative position of joints, the input is the connection vector of the inertial data of three sensors worn on the user's head joint, left wrist joint, and right wrist joint. A position estimation network is designed by making full use of past and future time information, expressed as b = S BN (X) An encoder combining Transformer Encoder and bidirectional long short-term memory network is designed to learn habitual data Based on the global and local time dependency deep features in the training data, a decoder based on a multi-layer perceptron is designed to estimate the positions of the three joints relative to the waist joint. The joint relative position loss of the head joint, left wrist joint, and right wrist joint is obtained through the predicted joint relative positions and the real joint relative positions, which is expressed as: in, is the predicted relative position of the joint, b GT is the real relative position of the joint; For the estimation of joint posture, the inertial data and the position of the joint relative to the waist joint are connected as As the first pose estimation network S PN1 (·) input, S PN1 (·) Predict the rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint in the 6D three-dimensional rotation representation and the global rotation of the waist joint The connected head joint, left elbow joint, and right elbow joint posture vectors can quickly obtain the posture information of the local parts; the loss function of the joint rotation is expressed as: Among them, the weight λ is set to 0.1, is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the head joint, left elbow joint, and right elbow joint relative to the parent joint, It is the real rotation of the head joint, left elbow joint and right elbow joint relative to the parent joint.

6. A whole body posture estimation method based on sparse inertial sensors according to claim 5, characterized in that: The position estimation network S BN (·) The structure consists of an encoder and a decoder: the encoder consists of a linear layer, a Transformer Encoder, and a bidirectional long short-term memory network, and the decoder consists of a linear layer.

7. A whole body posture estimation method based on sparse inertial sensors according to claim 5 or 6, characterized in that: The second stage is implemented as follows: The inertial data and the posture vectors of the head joint, waist joint, left elbow joint, and right elbow joint are used to estimate the posture of the upper body joints; the inertial data and the first posture estimation network S PN1 The output of (·) is connected in series X (2) Represents the input of the second stage; the second posture estimation network S PN2 (·) Output the rotation of the upper body joint relative to the parent joint and the global rotation of the waist joint The loss function of the connected upper body joint posture vector is expressed as: Among them, the weight λ is set to 0.1; is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the upper body joint relative to the parent joint, is the actual rotation of the upper body joint relative to the parent joint.

8. A whole body posture estimation method based on sparse inertial sensors according to claim 7, characterized in that: The third stage is implemented as follows: The inertial measurement values ​​and upper body joint posture vectors are used to estimate the posture of the whole body joints; the inertial data and the second posture estimation network S PN2 The output of (·) is connected in series X (3) Represents the input of the third stage; the third posture regression network S PN3 The output of (·) is the rotation of the whole body joint relative to the parent joint and the global rotation of the waist joint After the connection, the loss function of the whole body joint posture vector is calculated as: Among them, the weight λ is set to 0.1; is the predicted global rotation of the waist joint, is the global rotation of the real waist joint, is the predicted rotation of the whole body joint relative to the parent joint, It is the real rotation of the whole body joint relative to the parent joint.

9. A method for whole body posture estimation based on sparse inertial sensors according to claim 8, characterized in that: The second pose estimation network S PN2 (·) and the position estimation network S BN The structure of the third pose estimation network S is the same as that of PN3 (·) and the position estimation network S BN The structure of (·) is the same.

Images