Construction method of multi-stage joint position prediction network based on IMU (Inertial Measurement Unit)
By building a multi-stage joint position prediction network based on IMU, and using optical motion capture system and IMU data to align it, the data processing difficulties and performance instability caused by the complex IMU settings and the small number of IMUs in the prior art are solved, and the effect of predicting the joint position of the whole body is achieved through only two ear IMUs, reducing the difficulty of wearing sensors and improving application prospects.
Patent Information
- Application Number
- CN202510217129.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-26
- Publication Date
- 2025-06-03
AI Technical Summary
In the prior art, when using inertial sensors (IMUs) for three-dimensional motion reconstruction, complex IMU settings lead to difficulty in processing data, and the smaller the number of IMUs, the worse the performance, the unstable result, and the inability to effectively estimate global translation, resulting in the inability to present the body's motion process.
By constructing a multi-stage joint position prediction network based on IMU, using the optical motion capture system and data collected by the IMU for spatial and temporal alignment, and training a multi-stage neural network, it is possible to predict the whole-body joint position through only two ear IMUs.
It reduces the number and difficulty of wearing sensors, improves the application prospects in mobile environments, and broadens the applicable population.
Smart Images
Figure CN120078407A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of inertial sensors, and particularly relates to a method for constructing a multi-stage joint position prediction network based on an IMU. Background Art
[0002] Generally, trackers and sensors are worn on various parts of the pelvis or lower body to capture the human body posture during movement. However, these parts are difficult to wear, and the worn trackers and sensors will hinder the movement of the human body, which is extremely disadvantageous especially for patients with movement disorders and testers. Therefore, it is imperative to reduce the number of required sensors.
[0003] In recent years, significant progress has been made in methods for three-dimensional motion reconstruction using only inertial sensors (IMUs), including: 1) an extensible full-body estimation method that combines an HMD and a wearable IMU, but its complex IMU + HMD setup makes data processing very difficult; 2) DiffusionPoser can reconstruct human motion in real time at a rate of 25Hz from any IMU configuration for online motion reconstruction. However, as a generative model, the fewer IMUs, the worse its performance, resulting in very unstable results; 3) DynaIP uses sparse IMUs for real-time human pose estimation, attempting to find the optimal positions of the IMUs on the body to achieve the best pose estimation, but it does not estimate the global translation of the body, making it unable to present the movement process of the body. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a method for constructing a multi-stage joint position prediction network based on an IMU, which sequentially aligns the data collected by the optical motion capture system and the IMU in space and time. The real data collected by the optical motion capture system after alignment, that is, the MC sequence, is used as the reference sequence, and the measurement data collected by the IMU after alignment, that is, the IMU sequence, is used as the model input to train a multi-stage neural network. The finally obtained multi-stage joint position prediction network can achieve predicting the positions of joints throughout the body only through two ear-mounted IMUs, which reduces the number and difficulty of wearing sensors, improves its application prospects in a mobile environment, and broadens the applicable population.
[0005] This application provides a method for constructing a multi-stage joint position prediction network based on an IMU, including:
[0006] Collect gait data of a tester changing with time in a global coordinate system using an optical motion capture system, and represent it as an MC sequence. At the same time, collect inertial measurement data of the tester changing with time using IMUs worn on both ears.
[0007] Convert the inertial measurement data into binaural acceleration data in the global coordinate system, and represent the binaural acceleration data in the global coordinate system that changes over time as an IMU sequence;
[0008] Use the spatial semantic alignment method to perform time frame alignment processing on the IMU sequence and the MC sequence, so that the processed IMU sequence and MC sequence have the same spatio-temporal context;
[0009] Use the processed IMU sequence as the input and the MC sequence as the reference sequence, and use the standard LSTM to train a multi-stage neural network to obtain a multi-stage whole-body joint position prediction network.
[0010] Furthermore, the testers include: healthy individuals and patients with cervical spondylosis;
[0011] The method of using the optical motion capture system to collect the gait data of the tester that changes over time in the global coordinate system includes:
[0012] According to the Helen Hayes reflective marker placement method, wear reflective markers at each joint of the tester; wherein, the joints include: 6 trunk joints, 16 lower limb joints and 4 double-arm joints;
[0013] Use the optical motion capture system to collect the normal gait data of the healthy individual that changes over time, the simulated gait data of the healthy individual imitating the cervical spondylosis patient that changes over time, and the gait data of the cervical spondylosis patient that changes over time.
[0014] Furthermore, the inertial measurement data includes: acceleration relative to the IMU local coordinate system and direction relative to the IMU inertial coordinate system;
[0015] The step of converting the inertial measurement data into binaural acceleration data in the global coordinate system includes:
[0016] Based on the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system in the inertial measurement data, obtain the binaural acceleration relative to the IMU inertial coordinate system;
[0017] Based on the binaural acceleration relative to the IMU inertial coordinate system, use the conversion matrix from the IMU inertial coordinate system to the global coordinate system to obtain the predicted value of the binaural acceleration relative to the global coordinate system;
[0018] Based on the predicted value of the binaural acceleration relative to the global coordinate system and the binaural acceleration offset relative to the global coordinate system, obtain the binaural acceleration relative to the global coordinate system.
[0019] Furthermore, the conversion matrix from the IMU inertial coordinate system to the global coordinate system is obtained by the following method:
[0020] Fix the IMU on the central axis of the global coordinate system and place it perpendicular to the ground. At the same time, align the axes of the IMU local coordinate system with the corresponding axes of the global coordinate system;
[0021] After the IMU has been placed continuously for 3 s, read the initial measurement data output by the IMU;
[0022] Take the direction in the initial measurement data relative to the IMU inertial coordinate system as the transformation matrix from the IMU inertial coordinate system to the global coordinate system.
[0023] Further, obtain the binaural acceleration offset relative to the global coordinate system through the following method:
[0024] Wear the IMU on the ears of the tester respectively and keep it fixed. Then, after the tester's head and body are kept perpendicular and stationary for a predetermined time, read the secondary measurement data of the calibration process output by the IMU;
[0025] From the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system in the secondary measurement data, obtain the binaural acceleration relative to the IMU inertial coordinate system during the calibration process;
[0026] Based on the binaural acceleration relative to the IMU inertial coordinate system during the calibration process, use the transformation matrix from the IMU inertial coordinate system to the global coordinate system to obtain the predicted value of the binaural acceleration relative to the global coordinate system during the calibration process;
[0027] Take the predicted value of the binaural acceleration relative to the global coordinate system during the calibration process as the binaural acceleration offset relative to the global coordinate system.
[0028] Further, the time frame alignment process for the IMU sequence and the MC sequence using the spatial semantic alignment method includes:
[0029] Align the IMU sequence and the MC sequence according to the initial time point;
[0030] Interpolate the missing positions in the aligned IMU sequence to make the frame rate of the IMU sequence the same as that of the MC sequence;
[0031] Perform redundant data removal processing on the interpolated IMU sequence.
[0032] Further, taking the processed IMU sequence as the input and the MC sequence as the reference sequence, training a multi-stage neural network using a standard LSTM to obtain a multi-stage full-body joint position prediction network includes:
[0033] Use the processed IMU sequence as the input of a multi-stage neural network, and use standard LSTM to sequentially learn the multi-stage mapping relationships from binaural acceleration to torso joint positions, from torso joint positions to lower limb joint positions, and from lower limb joint positions to upper limb joint positions. Use the MC sequence as the reference sequence to update the model parameters, so as to obtain a multi-stage full-body joint position prediction network that can predict full-body joint positions based on binaural acceleration.
[0034] Further, before training the multi-stage neural network, the method further includes:
[0035] Convert the joint positions of the tester from the global coordinate system to the body coordinate system centered on the root node. Specifically:
[0036] Use an optical motion capture system to capture the predetermined pose images of the tester to obtain the human skeletal information of the tester;
[0037] Based on the human skeletal information, determine the initial positions of the full-body joints of the tester relative to the heel joint. The method for constructing a multi-stage joint position prediction network based on IMU provided by this application aligns the data collected by the optical motion capture system and IMU in space and time in sequence. Use the real data collected by the optical motion capture system after alignment, that is, the MC sequence as the reference sequence, and use the measurement data collected by the IMU after alignment, that is, the IMU sequence as the model input to train the multi-stage neural network. The finally obtained multi-stage joint position prediction network can realize predicting full-body joint positions only through two ear IMUs, which reduces the number and difficulty of wearing sensors, and improves its application prospects in mobile environments, as well as broadens the applicable population. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 Shows a flowchart of a method for constructing a multi-stage joint position prediction network based on IMU provided by an embodiment of the present application;
[0039] Figure 2 Shows a diagram of human full-body joint positions and region divisions provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0040] To make the objectives, technical solutions, and advantages of the technical solution clearer, the technical solution will be further described in detail below in conjunction with specific embodiments. It should be understood that these descriptions are exemplary and not intended to limit the scope of the technical solution.
[0041] Please refer to Figure 1 , Figure 1 shown is a flowchart of a method for constructing a multi-stage joint position prediction network based on IMU provided by an embodiment of the present application. As Figure 1As shown, the method includes:
[0042] S101. Use an optical motion capture system to collect the gait data of the tester changing with time in the global coordinate system, and represent it as an MC sequence. At the same time, use the IMUs worn on both ears to collect the inertial measurement data of the tester changing with time.
[0043] Among them, the tester includes: healthy individuals and patients with cervical spondylosis.
[0044] As an example, the tester includes a total of 10 healthy individuals and 1 patient with cervical spondylosis. The patient with cervical spondylosis here can be a patient with spondylotic myelopathy. The optical motion capture system is a NOKOV commercial-grade optical motion capture device, which includes 12 high-precision infrared cameras, 1 data switch, and an optical motion capture processing system.
[0045] In specific implementation, the following method is used to collect the gait data of the tester changing with time in the global coordinate system by using the optical motion capture system:
[0046] Step 1011. According to the Helen Hayes reflective marker placement method, wear reflective markers at each joint of the tester.
[0047] Among them, the joints include: 6 trunk joints, 16 lower limb joints, and 4 bilateral arm joints.
[0048] In this step, according to the Helen Hayes reflective marker placement method, wear reflective markers at each joint of the tester, and use nylon fasteners to fix the reflective markers on each joint. Specifically, please refer to Figure 2 the human body whole-joint position and area division diagram provided in the embodiment of the present application as shown. Figure 2 In it, the back, sacrum, left and right waists are divided into trunk joints, the left and right thighs, left and right inner knees, left and right outer knees, left and right calves, left and right inner ankles, left and right outer ankles, left and right heels, and left and right toes are divided into lower limb joints, and the left and right wrists, left and right elbows, and left and right shoulders are divided into bilateral arm joints.
[0049] Step 1012. Use the optical motion capture system to collect the normal gait data of the healthy individual changing with time, the simulated gait data of the healthy individual imitating the patient with cervical spondylosis changing with time, and the gait data of the patient with cervical spondylosis changing with time.
[0050] In this step, for healthy individuals, gait data under two conditions need to be collected. First, let the healthy individual walk normally in a comfortable posture to collect the normal gait data of the healthy individual changing over time. Then, let the healthy individual walk imitating the posture of a cervical spondylosis patient to collect the simulated gait data of the healthy individual imitating the cervical spondylosis patient changing over time. As an example, the ratio of the normal gait data to the simulated gait data of the healthy individual is 2:1. For cervical spondylosis patients, their walking postures are not restricted, that is, the gait data of the cervical spondylosis patients changing over time in the natural state is collected. Among them, the optical motion capture system collects the gait data of the tester at a frame rate of 60 FPS.
[0051] S102. Convert the inertial measurement data into binaural acceleration data in the global coordinate system, and represent the binaural acceleration data in the global coordinate system changing over time as an IMU sequence.
[0052] Among them, the inertial measurement data includes: the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system.
[0053] In this step, since the inertial measurement data and the gait data are collected using different acquisition devices, and each acquisition device has its own coordinate system, then different data needs to be converted into the same coordinate system. Due to the fixed nature of the optical motion capture system, in the embodiments of the present application, the inertial measurement data is converted into the global coordinate system.
[0054] In specific implementation, the inertial measurement data is converted into binaural acceleration data in the global coordinate system through the following method:
[0055] Step 1021. Based on the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system in the inertial measurement data, obtain the binaural acceleration relative to the IMU inertial coordinate system.
[0056] In this step, the coordinate system of each IMU is defined as the IMU local coordinate system F S , and the coordinate system formed by two IMUs worn on both ears is defined as the IMU inertial coordinate system F I . The inertial measurement data output by the IMU includes the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system Based on the above two components, the acceleration in the IMU local coordinate system can be
[0057] converted into the IMU inertial coordinate system.
[0058] As an example, the binaural acceleration relative to the IMU inertial coordinate system can be obtained through the following formula (1)
[0059]
[0060] Step 1022: Based on the binaural accelerations with respect to the IMU inertial coordinate system, use the transformation matrix from the IMU inertial coordinate system to the global coordinate system to obtain the predicted values of the binaural accelerations with respect to the global coordinate system.
[0061] In this step, due to the fixed nature of the optical motion capture system, the coordinate system of the optical motion capture system is defined as the global coordinate system F M . Here, the pre-determined transformation matrix P IM from the IMU inertial coordinate system to the global coordinate system is used to transform the binaural accelerations in the IMU inertial coordinate system
[0062] to the global coordinate system. As an example, the predicted values of the binaural accelerations with respect to the global coordinate system can be obtained through the following formula (2)
[0063]
[0064] In specific implementation, the transformation matrix from the IMU inertial coordinate system to the global coordinate system is obtained in the following manner:
[0065] Step 201: Fix the IMU on the central axis of the global coordinate system and place it perpendicular to the ground. At the same time, align each axis of the IMU local coordinate system with the corresponding axes of the global coordinate system.
[0066] In this step, the IMU is fixed on the central axis of the global coordinate system, with the X-axis of the IMU local coordinate system pointing to the right, the Y-axis pointing forward, and the Z-axis pointing upward, so as to align the IMU local coordinate system with the global coordinate system.
[0067] Step 202: After the IMU has been placed continuously for 3 s, read the initial measurement data output by the IMU.
[0068] In this step, normally, similar to the inertial measurement data, the initial measurement data also includes the acceleration with respect to the IMU local coordinate system and the orientation with respect to the IMU inertial coordinate system. However, due to the fixation of the IMU, the acceleration with respect to the IMU local coordinate system is 0. Therefore, the initial measurement data only includes the orientation with respect to the IMU inertial coordinate system.
[0069] Step 203: Use the orientation with respect to the IMU inertial coordinate system in the initial measurement data as the transformation matrix from the IMU inertial coordinate system to the global coordinate system.
[0070] In this step, since the local coordinate system of the IMU has been aligned with the global coordinate system, the direction of the initial measurement data relative to the inertial coordinate system of the IMU can be used as the transformation matrix from the inertial coordinate system of the IMU to the global coordinate system.
[0071] Step 1023: Based on the predicted values of binaural acceleration relative to the global coordinate system and the binaural acceleration offset relative to the global coordinate system, obtain the binaural acceleration relative to the global coordinate system.
[0072] In this step, due to sensor errors and the uncertainty of the transformation matrix, there will be an offset in the process of converting inertial measurement data to the global coordinate system. Therefore, the binaural acceleration offset of the global coordinate system needs to be considered.
[0073] As an example, the binaural acceleration relative to the global coordinate system can be obtained through the following formula (3).
[0074]
[0075]
[0076] In specific implementation, the binaural acceleration offset relative to the global coordinate system is obtained in the following way:
[0077] Step 301: Wear the IMUs on the ears of the tester respectively and keep them fixed. Then, keep the tester's head perpendicular to the body and stationary for a predetermined time, and read the secondary measurement data of the calibration process output by the IMUs.
[0078] In this step, it is actually to calibrate the inertial coordinate system of the IMU. Normally, similar to the inertial measurement data, the secondary measurement data also includes the acceleration relative to the local coordinate system of the IMU and the direction relative to the inertial coordinate system of the IMU.
[0079] Step 302: Obtain the binaural acceleration relative to the inertial coordinate system of the IMU during the calibration process from the acceleration relative to the local coordinate system of the IMU and the direction relative to the inertial coordinate system of the IMU in the secondary measurement data.
[0080] In this step, the binaural acceleration relative to the inertial coordinate system of the IMU during the calibration process is obtained using the above formula (1).
[0081] Step 303: Based on the binaural acceleration relative to the inertial coordinate system of the IMU during the calibration process, use the transformation matrix from the inertial coordinate system of the IMU to the global coordinate system to obtain the predicted value of the binaural acceleration relative to the global coordinate system during the calibration process.
[0082] In this step, the predicted binaural acceleration values with respect to the global coordinate system during the calibration process are obtained using the above formula (2).
[0083] Step 304: Use the predicted binaural acceleration values with respect to the global coordinate system during the calibration process as the binaural acceleration offset with respect to the global coordinate system.
[0084] In this step, since the head of the tester remains perpendicular and stationary to the body during the calibration process, the IMU inertial coordinate system and the body coordinate system remain unchanged. Theoretically, the predicted binaural acceleration values with respect to the global coordinate system should be 0. Therefore, the obtained predicted binaural acceleration values with respect to the global coordinate system during the calibration process are the binaural acceleration offsets with respect to the global coordinate system.
[0085] S103: Use the spatial semantic alignment method to perform time-frame alignment processing on the IMU sequence and the MC sequence so that the processed IMU sequence and MC sequence have the same spatio-temporal context.
[0086] In this step, since the optical motion capture system collects the gait data of the tester at a frame rate of 60 FPS, that is, the frame rate of the MC sequence is 60, while the frame rate of the IMU sequence is 200. Therefore, it is necessary to align the IMU sequence and the MC sequence spatio-temporally.
[0087] In specific implementation, the following method is used to perform time-frame alignment processing on the IMU sequence and the MC sequence:
[0088] Step 1031: Align the IMU sequence with the MC sequence according to the initial time point.
[0089] Step 1032: Interpolate the missing positions in the aligned IMU sequence so that the frame rate of the IMU sequence is the same as that of the MC sequence.
[0090] In this step, the spline interpolation algorithm is used to align the IMU sequence and the MC sequence through the following formula (4):
[0091] S(a) = w + x(a - a j ) + y(a - a j ) 2 + z(a - a j ) 3 ; (4)
[0092] In the formula, S(a) is the interpolated IMU sequence, a is the acceleration at the missing position, a j is the acceleration at the known position, and w, x, y, z are the sparsities of the spline function.
[0093] Step 1033: Process the interpolated IMU sequence to remove redundant data.
[0094] S104: Use the processed IMU sequence as the input and the MC sequence as the reference sequence, and train a multi-stage neural network using a standard LSTM to obtain a multi-stage full-body joint position prediction network.
[0095] In a specific implementation, the multi-stage full-body joint position prediction network is obtained in the following manner:
[0096] Use the processed IMU sequence as the input of the multi-stage neural network, and use a standard LSTM to sequentially learn the multi-stage mapping relationships from the binaural acceleration to the trunk joint positions, from the trunk joint positions to the lower limb joint positions, and from the lower limb joint positions to the upper limb joint positions. Use the MC sequence as the reference sequence to update the model parameters to obtain a multi-stage full-body joint position prediction network that can predict the full-body joint positions based on the binaural acceleration. In this step, the multi-stage neural network includes a first-stage neural network, a second-stage neural network, and a third-stage neural network; specifically, the multi-stage neural network is trained in the following manner:
[0097] Step 1041: Input the processed IMU sequence into the first-stage neural network, use a standard LSTM to learn the mapping from the binaural acceleration to the trunk joint positions, output the increments of each trunk joint, and obtain the loss of the trunk joint positions according to the MC sequence to update the first-stage neural network.
[0098] In this step, the loss of the trunk joint positions is obtained through the following formula (5)
[0099]
[0100] where p T (t) is the increment of each trunk joint output by the first-stage neural network, is the increment of each trunk joint obtained from the MC sequence.
[0101] Step 1042: Input the processed IMU sequence and the increments of each trunk joint into the second-stage neural network, use a standard LSTM to learn the mapping from the trunk joint positions to the lower limb joint positions, output the increments of each lower limb joint, and obtain the loss of the lower limb joint positions according to the MC sequence to update the second-stage neural network.
[0102] In this step, the loss of the lower limb joint positions is obtained through the following formula (6)
[0103]
[0104] Where p L (t) is the increment of each lower limb joint output by the neural network in the second stage, and is the increment of each lower limb joint obtained from the MC sequence.
[0105] Step 1043: Input the processed IMU sequence and the increment of each lower limb joint into the neural network in the third stage, and use standard LSTM to learn the mapping from the lower limb joint position to the upper limb joint position, so as to output the increment of each upper limb joint, and obtain the loss of the upper limb joint position according to the MC sequence, so as to update the neural network in the third stage.
[0106] In this step, the loss of the upper limb joint position is obtained through the following formula (7)
[0107]
[0108] Where p U (t) is the increment of each upper limb joint output by the neural network in the third stage, and is the increment of each upper limb joint obtained from the MC sequence.
[0109] Furthermore, the total loss function of the multi-stage neural network is defined as:
[0110]
[0111] Where α, β, and γ are the weight parameters corresponding to the loss of the trunk joint position, the loss of the lower limb joint position, and the loss of the upper limb joint position respectively.
[0112] In addition, before training the multi-stage neural network, the method further includes:
[0113] Convert the joint position of the tester from the global coordinate system to the body coordinate system centered on the root node. Specifically:
[0114] Step 501: Use an optical motion capture system to capture the predetermined pose image of the tester to obtain the human skeleton information of the tester.
[0115] In this step, since the bone lengths and joint positions of different testers are different, for each tester, it is necessary to capture the predetermined pose image and use the OpenPose method to obtain the human skeleton information of the tester.
[0116] Step 502: Based on the human skeleton information, determine the initial position of the whole body joints of the tester relative to the heel joint.
[0117] In this step, based on the human bone information, SMPLify-X is used to generate a matching 3D human virtual model for the tester, so as to obtain the initial positions of the tester's whole-body joints relative to the heel joint.
[0118] The above content is only the preferred embodiment of the present invention. For those of ordinary skill in the art, many changes can be made in the specific implementation manner and application scope according to the idea of the present technical content. As long as these changes do not depart from the concept of the present invention, they all fall within the protection scope of this patent.
Claims
1. A method for constructing a multi-stage joint position prediction network based on IMU, characterized in that: The method comprises: An optical motion capture system is used to collect the tester's gait data that changes over time in the global coordinate system and is expressed as an MC sequence. At the same time, an IMU worn on both ears is used to collect the tester's inertial measurement data that changes over time. The inertial measurement data is converted into binaural acceleration data in a global coordinate system, and the binaural acceleration data in the global coordinate system that changes with time is represented as an IMU sequence; Using a spatial semantic alignment method, the IMU sequence and the MC sequence are subjected to time frame alignment processing, so that the processed IMU sequence and the MC sequence have the same spatiotemporal context; The processed IMU sequence is used as input and the MC sequence is used as the reference sequence. A multi-stage neural network is trained using the standard LSTM to obtain a multi-stage whole-body joint position prediction network.
2. The method according to claim 1, characterized in that The test subjects include: healthy individuals and patients with cervical spondylosis; The method of using an optical motion capture system to collect the gait data of the tester that changes over time in a global coordinate system includes: According to the Helen Hayes reflective marker placement method, reflective markers are worn on the joints of the test subject; wherein the joints include: 6 trunk joints, 16 lower limb joints and 4 arm joints; The optical motion capture system is used to collect the normal gait data of the healthy individual that changes over time, the simulated gait data of the healthy individual that imitates the cervical spondylosis patient that changes over time, and the gait data of the cervical spondylosis patient that changes over time.
3. The method according to claim 1, characterized in that The inertial measurement data includes: acceleration relative to the IMU local coordinate system and direction relative to the IMU inertial coordinate system; The step of converting the inertial measurement data into binaural acceleration data in a global coordinate system comprises: Based on the acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system in the inertial measurement data, the binaural acceleration relative to the IMU inertial coordinate system is obtained; Based on the binaural acceleration relative to the IMU inertial coordinate system, the predicted binaural acceleration relative to the global coordinate system is obtained using the transformation matrix from the IMU inertial coordinate system to the global coordinate system; The binaural acceleration relative to the global coordinate system is obtained based on the binaural acceleration prediction value relative to the global coordinate system and the binaural acceleration offset relative to the global coordinate system.
4. The method according to claim 3, characterized in that The transformation matrix from the IMU inertial coordinate system to the global coordinate system is obtained by the following method: The IMU is fixed on the central axis of the global coordinate system and placed perpendicular to the ground, and the axes of the IMU local coordinate system are aligned with the corresponding axes of the global coordinate system; After the IMU is placed for 3 seconds, read the initial measurement data output by the IMU; The direction of the initial measurement data relative to the IMU inertial coordinate system is used as a transformation matrix from the IMU inertial coordinate system to the global coordinate system.
5. The method according to claim 3, characterized in that The binaural acceleration offset relative to the global coordinate system is obtained by the following method: The IMUs are respectively worn on the tester's ears and kept fixed, and then the tester's head is kept vertical to the body and remains still for a predetermined time, and then the secondary measurement data of the calibration process output by the IMU is read; The acceleration relative to the IMU local coordinate system and the direction relative to the IMU inertial coordinate system in the secondary measurement data are obtained to obtain binaural acceleration relative to the IMU inertial coordinate system during the calibration process; Based on the binaural accelerations relative to the IMU inertial coordinate system during the calibration process, a predicted value of the binaural accelerations relative to the global coordinate system during the calibration process is obtained using a transformation matrix from the IMU inertial coordinate system to the global coordinate system; The binaural acceleration prediction value relative to the global coordinate system in the calibration process is used as the binaural acceleration offset relative to the global coordinate system.
6. The method according to claim 1, characterized in that The method of using a spatial semantic alignment method to perform time frame alignment processing on the IMU sequence and the MC sequence includes: According to the initial time point, aligning the IMU sequence with the MC sequence; Interpolating the missing positions of the aligned IMU sequence so that the frame rate of the IMU sequence is the same as the frame rate of the MC sequence; The redundant data is eliminated for the interpolated IMU sequence.
7. The method according to claim 1, characterized in that The processed IMU sequence is used as input, the MC sequence is used as a reference sequence, and a multi-stage neural network is trained using a standard LSTM to obtain a multi-stage whole body joint position prediction network, including: The processed IMU sequence is used as the input of the multi-stage neural network. The standard LSTM is used to sequentially learn the multi-stage mapping relationships from binaural acceleration to trunk joint position, from trunk joint position to lower limb joint position, and from lower limb joint position to arm joint position. The MC sequence is used as the reference sequence to update the model parameters to obtain a multi-stage whole-body joint position prediction network that can predict whole-body joint positions based on binaural acceleration.
8. The method according to claim 7, characterized in that Prior to training the multi-stage neural network, the method further comprises: The joint positions of the test subject are converted from the global coordinate system to the body coordinate system centered on the root node, specifically: Using an optical motion capture system to capture a predetermined posture image of the tester to obtain human skeleton information of the tester; Based on the human skeleton information, the initial positions of the tester's whole body joints relative to the heel joint are determined.