Method and system for predicting whole body actions by sparse inertial measurement unit

Abstract

The invention discloses a method and a system for predicting whole body actions by a sparse inertial measurement unit, and the method comprises the following steps: setting a plurality of nodes at a test target, and setting inertial measurement modules on the nodes; acquiring a node inertia action signal and recording a message sequence and a timestamp of the message; acquiring a node rotation attitude and a node acceleration in a geographic coordinate system; acquiring node rotation attitudes and node accelerations of a plurality of nodes under the SMPL coordinate system; calculating the positions of all leaf nodes relative to the root node; calculating the position of the whole body joint relative to the root node; calculating to obtain a rotating posture of the whole-body joint relative to the root node; and measuring the speed of the root node. According to the technical scheme provided by the invention, the non-deployed IMU joint action can be more accurately predicted, the high-speed action can be captured, the long-time coherent action restoration is realized, the real action and virtual action delay is shortened, the system provided by the invention is low in cost, and frequent action calibration is not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion capture, and in particular relates to a method and system for predicting whole-body motion using a sparse inertial measurement unit. Background Art

[0002] Motion capture has a wide range of application scenarios, such as motion monitoring, film production, medical rehabilitation, gaming entertainment, social networking, etc. The mainstream technologies of existing motion capture are based on optics and inertial measurement units (IMU). Among them, optical motion capture completes the task of motion capture by monitoring and tracking specific markers on the target. Taking Vicon technology as an example, in the actual motion capture process, it is necessary to deploy optical markers on the joints of the subject, and then specify the capture area and arrange multiple high-speed cameras around the capture area to capture the optical markers. Usually, infrared light source high-speed cameras are used, and the optical markers need to be captured by more than two cameras at the same time. Motion capture technology based on inertial measurement units (IMUs), such as wearable motion sensors Xsens, deploys IMUs on the joints of the subject to capture and record motion data, and performs motion measurements such as limb acceleration, speed, and direction changes; the motion capture product mocopi released by Sony (China) Co., Ltd. consists of 6 sensors worn on the limbs, head and waist with applications, which can realize real-time motion capture and data processing.

[0003] However, optical motion capture requires expensive cameras, has high requirements for light and environment, has a complex preparation process before capture, has a limited capture area, is difficult to apply to large-scale scenes, and invasive optical markers interfere with the subject's movements. These problems make optical motion capture unsuitable for consumer applications. Among existing motion capture systems based on inertial measurement units, Xsens uses 17 IMUs to capture the rotation of human joints to obtain more accurate human movements, but densely placed IMUs interfere with the subject's movements and lead to high costs; mocopi uses fewer IMUs to capture the rotation of the head, waist, forearms on both sides, and calves on both sides, and then infers the rotation of other joints that do not deploy IMUs. Although this can reduce costs, mocopi's technical solution fails to solve the problem of IMU drift, so it takes more than ten minutes to perform a motion calibration, and the accuracy of the rotation prediction of joints that do not deploy IMUs is poor, and there is a significant delay in motion restoration, which affects the user experience.

[0004] Patent number CN118097775A discloses a human body shape-aware sparse IMU motion capture method and system, which obtains the IMU inertial measurement value of the human body, inputs the IMU inertial measurement value into a recurrent neural network to solve the human posture to estimate the human motion posture, and then uses the recurrent neural network to optimize the human motion posture to obtain the human motion capture result. This method and system considers the use of multiple inertial measurement units (IMUs) for real-time human motion capture, and can capture the movements of people of different body shapes, but the problems of high-speed motion capture, long-term continuous motion restoration, and delay have not been solved, and the accuracy of motion prediction for joints without IMU deployment cannot be improved. Summary of the invention

[0005] The purpose of the present invention is to provide a method for predicting whole-body motion using a sparse inertial measurement unit, which can more accurately predict undeployed IMU joint motions, capture high-speed motions, achieve long-term continuous motion restoration, and shorten the delay between real motion and virtual motion.

[0006] Another object of the present invention is to provide a system for predicting whole-body motion using a sparse inertial measurement unit that can more accurately predict undeployed IMU joint motions, capture high-speed motions, achieve long-term continuous motion restoration, shorten the delay between real and virtual motions, and does not require frequent motion calibration and is low-cost.

[0007] The method for predicting whole-body motion using a sparse inertial measurement unit provided by the present invention comprises:

[0008] Step S1, setting a plurality of nodes on a test target, including a root node and a plurality of leaf nodes, and setting an inertial measurement module on the nodes;

[0009] Step S2, obtaining node inertial motion signals measured by multiple inertial measurement modules, and recording the message sequence and message timestamp of the node inertial motion signals;

[0010] Step S3, preprocessing the multiple node inertial motion signals to obtain node inertial data measurement values, fusing the node inertial data measurement values ​​to obtain node rotation posture and node acceleration in a geographic coordinate system;

[0011] Step S4, converting the node rotation attitude and node acceleration of the geographic coordinate system into the SMPL coordinate system, and obtaining the node rotation attitude and node acceleration of multiple nodes in the SMPL coordinate system;

[0012] Step S5, the node rotation postures and node accelerations of multiple nodes in the SMPL coordinate system are synchronously sent to the neural network to calculate the positions of all leaf nodes relative to the root node;

[0013] Step S6, the position of the leaf node relative to the root node, the rotation posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the position of the whole body joints relative to the root node;

[0014] Step S7, the positions of the joints of the whole body relative to the root node, the rotational posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the rotational posture of the joints of the whole body relative to the root node;

[0015] In step S8, the node rotation postures and node accelerations of multiple nodes are synchronously sent to the bidirectional recurrent neural network to estimate the probability of landing on the left and right sides, and the higher value of the left and right landing probabilities is used as the effective value of the landing probability to calculate the speed of the root node.

[0016] Preferably, the multiple nodes in step S1 are set to 6, wherein the node behind the waist is set as the root node, the nodes near the wrist joints of the left and right forearms, the nodes near the knee joints of the left and right calves, and the node behind the head are set as leaf nodes.

[0017] Preferably, the synchronization algorithm of step S5 synchronously sends the node rotation posture and node acceleration in the SMPL coordinate system with consistent cached message order to the neural network.

[0018] Preferably, the training of the neural network in step S4 includes an initial calibration action, processing the action error using the SMPL coordinate system, converting the node rotation attitude and node acceleration in the geographic coordinate system to the SMPL coordinate system, and outputting the node rotation attitude and node acceleration in the SMPL coordinate system.

[0019] Preferably, in step S5, a bidirectional recurrent neural network is used to estimate the position of leaf nodes relative to the root node, in step S6, a bidirectional recurrent neural network is used to estimate the position of whole body joints relative to the root joint, and in step S7, a bidirectional recurrent neural network is used to estimate the rotational posture of whole body joints relative to the root node.

[0020] Preferably, the speed of the root node is calculated in step S8 in the following manner: a threshold range of the landing probability is preset, and if the effective value of the landing probability is less than the minimum value of the preset threshold range, the speed of the root node is predicted using the RNN recurrent neural network and recorded as the first speed; if the effective value of the landing probability is greater than the maximum value of the preset threshold range, the speed of the root node is calculated using forward dynamics and recorded as the second speed; if the effective value of the landing probability is within the preset threshold range, the speed of the root node at this time is calculated by combining the first speed predicted by the RNN recurrent neural network and the second speed calculated using forward dynamics and recorded as the third speed.

[0021] Another aspect of the present invention provides a system for predicting whole-body motion using a sparse inertial measurement unit, comprising an inertial measurement module, an auxiliary navigation module, a communication module, and an operation display module, wherein:

[0022] There are multiple inertial measurement modules, which are used to be deployed at different node positions of the human body, capture node inertial motion signals, and pre-process the node inertial motion signals into node inertial data measurement values;

[0023] The auxiliary navigation module is used to fuse the node inertial data measurement values ​​to obtain the joint rotation postures and joint accelerations of multiple nodes in the SMPL coordinate system;

[0024] The communication module is used to obtain the joint rotation postures and joint accelerations of multiple nodes in the SMPL coordinate system, and synchronously send them to the operation display module;

[0025] The operation display module executes the above-mentioned method of predicting whole-body movements by the sparse inertial measurement unit to predict the whole-body posture.

[0026] Preferably, the inertial measurement module includes a microcontroller unit MCU, an inertial measurement unit IMU, a magnetometer, a wireless communication unit and a battery unit. The inertial measurement unit IMU and the magnetometer are used to capture node inertial motion signals. The microcontroller unit MCU collects the node inertial motion signals captured by the inertial measurement unit IMU and the magnetometer and performs preprocessing.

[0027] Preferably, the microcontroller unit MCU drives the inertial measurement unit IMU and the magnetometer through serial communication, and obtains the node inertial motion signal; the microcontroller unit MCU drives the wireless communication unit through serial communication.

[0028] Preferably, there are at least six inertial measurement modules, which are respectively deployed behind the waist of the test target, behind the head, near the wrist joints of the left and right forearms, and near the knee joints of the left and right calves.

[0029] Preferably, the inertial measurement module uses an extended Kalman filter EKF to preprocess the node inertial motion signal to obtain the node inertial data measurement value, and the auxiliary navigation module is an attitude reference system AHRS, through which the node inertial motion signal is fused to obtain the node rotation attitude and node acceleration.

[0030] Preferably, the communication module is a router, the wireless communication unit is a Wifi module, the operation display module is a computer, the Wifi module synchronizes the multiple inertial measurement units IMU with the clock of the computer, and the router synchronously sends the node rotation postures and node accelerations of the multiple nodes to the operation display module through a network communication protocol.

[0031] Preferably, the operation display module is a computer or intelligent terminal including a neural network algorithm.

[0032] Compared with the prior art, the technical solution protected by the present invention has the following advantages:

[0033] The present invention adopts a data synchronization algorithm, pre-processes the inertial motion signal, obtains the inertial data measurement value and caches it, uses the heading reference system to fuse the node inertial motion signal, obtains the node rotation attitude and node acceleration, and sends the node inertial data measurement value, node rotation attitude and node acceleration with consistent cached message order to the trained neural network synchronously, thereby improving the accuracy of motion restoration, being able to capture high-speed motion, realize long-term coherent motion restoration, and shorten the delay between real motion and virtual motion. The neural network algorithm adopted by the present invention can more accurately predict the undeployed IMU joint motion.

[0034] The system provided by the present invention uses an inertial measurement module and an auxiliary navigation module to predict human posture, uses a magnetometer to correct the heading angle in real time, and uses an EKF algorithm to compensate for the zero bias, scale factor, and non-orthogonal deviation of the magnetometer in real time, thereby avoiding frequent calibration. In addition, the communication module of the present invention uses a router to form a network, thereby ensuring the stability and real-time performance of the communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 Schematic diagram of system modules for predicting whole-body motion using a sparse inertial measurement unit according to an embodiment of the present invention.

[0037] Figure 2 Schematic diagram of the deployment of the inertial measurement module in an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the function of the inertial measurement module in an embodiment of the present invention.

[0039] Figure 4Flowchart of our method for predicting full-body motion for a sparse inertial measurement unit.

[0040] Figure 5 Schematic diagram of the synchronization algorithm in an embodiment of the present invention.

[0041] Figure 6 Schematic diagram of the whole body posture prediction process in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The whole body posture prediction system and prediction method proposed by the present invention are as follows: Figure 1 The above is a system module diagram of an embodiment of the present invention, including an inertial measurement module, an auxiliary navigation module 20, a communication module 30 and an operation display module 40. There are multiple inertial measurement modules. In this embodiment, there are 6 measurement modules, which are respectively arranged on 6 main joints to capture the node inertial motion signals of these joints and pre-process the node inertial motion signals into inertial data measurement values; the auxiliary navigation module 20 fuses the node inertial motion signals to obtain the node rotation posture and node acceleration in the SMPL coordinate system; the communication module 30 obtains the node rotation posture and node acceleration, and sends them to the operation display module 40; the operation display module 40 processes and displays the obtained data.

[0044] like Figure 2As shown, it is the layout diagram of the inertial measurement module in the human body. Among them, the inertial measurement module 11 and the inertial measurement module 12 are deployed near the wrist joints of the left and right forearms, the inertial measurement module 13 and the inertial measurement module 14 are deployed near the knee joints of the left and right calves, and the inertial measurement module 15 and the inertial measurement module 16 are deployed behind the head and behind the waist to measure the inertial signals of the above nodes. Among them, the node behind the waist is the root node, and the data is obtained by the inertial measurement module 16, and the nodes at other positions are leaf nodes. By establishing a virtual human body model containing 24 joints of the human body, other joints where the measurement modules are not deployed rely on inertial navigation, human kinematics model, and neural network to complete their rotation postures. In this embodiment, the 24 joints of the human body are: left hip (Left_Hip), right hip (Right_Hip), waist (Waist), left knee (Left_Knee), right knee (Right_Knee), upper waist (Upper_Waist), left ankle (Left_Ankle), right ankle (Right_Ankle), chest (Chest), left toe (Left_Toe), right toe (Right_Toe), base of neck (Base_Neck), left shoulder (Left_Shoulder), right shoulder (Right_Shoulder), upper neck (Upper_Neck), left arm (Left_Arm), right arm (Right_Arm), left elbow (Left_Elbow), right elbow (Right_Elbow), left wrist (Left_Wrist), right wrist (Right_Wrist), left finger (Left_Finger), and right finger (Right_Finger).

[0045] Further preferred embodiments, such as Figure 3 As shown, the inertial measurement module 11 includes a microcontroller unit MCU111, an inertial measurement unit IMU112, a magnetometer 113, a wireless communication unit and a battery unit. The inertial measurement unit IMU112 and the magnetometer 113 are used to capture the inertial motion signal of the human body node. The microcontroller unit MCU111 collects the inertial motion signal of the human body node captured by the inertial measurement unit IMU112 and the magnetometer 113, and pre-processes the inertial motion signal of the node. The other measurement data modules adopt the same scheme.

[0046] In this embodiment, an extended Kalman filter EKF is used to pre-process the node inertial motion signal, and the auxiliary navigation module 20 uses the attitude reference system AHRS to fuse the node inertial motion signal to obtain the node rotation attitude and node acceleration in the SMPL coordinate system, such as Euler angles, quaternions or rotation matrices, and sends the node rotation attitude and node acceleration in the SMPL coordinate system to the communication module 30.

[0047] In this embodiment, the inertial measurement unit IMU is used to measure the angular velocity and acceleration of the object, and includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometer is used to detect the three-axis acceleration signal of the object in the carrier coordinate system, and the gyroscope is used to measure the angular velocity signal of the object relative to the navigation coordinate system, so as to obtain the angular velocity and acceleration of the object in three-dimensional space, so as to solve the attitude of the object. In addition, the inertial measurement unit IMU also includes a three-axis magnetometer, which measures the component of the earth's magnetic field in the object coordinate system to determine the heading of the object. By fusing the data of accelerometers, gyroscopes and magnetometers, the inertial measurement unit IMU can play an important role in navigation, attitude control and other fields.

[0048] In this embodiment, the attitude calculation algorithm (AHRS algorithm) of the inertial measurement unit IMU is the Mahony algorithm, and the process is as follows:

[0049] The geographic coordinate system uses the Northeast Sky coordinate system, marked as E, and the human body coordinate system uses the SMPL coordinate system, marked as S. The data returned by the IMU's three-axis gyroscope is the angular velocity of the system around the x-axis, y-axis, and z-axis. These three angular velocities are represented by ω x ,ω y ,ω z The data returned by the gyroscope can be regarded as a quaternion with a real part of zero, expressed as S ω is used to represent:

[0050]

[0051] Attitude quaternion change speed With current posture and angular velocity S ω related:

[0052]

[0053] The quaternion at time t-1 is known and angular velocity S ω t-1 , and the angular velocity at time t S ω t , the system sampling interval is Δt, find the quaternion at time t Similar to ordinary differential equations, the improved Euler formula is applied:

[0054]

[0055] Approximately estimate the average change rate of the object's quaternion from time t-1 to time t In practice, we often cannot get the exact quaternion at time t-1, but an optimal estimate. then:

[0056]

[0057] Assuming that the gyroscope calculates the attitude accurately, then the gravity vector E is In the S system, it should be:

[0058]

[0059] Acceleration vector for attitude solution S g t and the actual accelerometer measured The deviation is:

[0060] S e a,t =cross( S a t , S g t )

[0061] Assume that the measurement value of the magnetometer in the S system is After rotating to E system:

[0062]

[0063] Since the E system is the northeast celestial coordinate system, Magnetic lines of force only have components along the x and z axes.

[0064] Theoretically y ≈0, but the actual attitude solution has an error h y ≠0, so:

[0065]

[0066] After processing E b t Rotate to S system, and we get s W t :

[0067]

[0068] Then calculate and S series measurement value S m t Deviation:

[0069] S e m,t =cross( S m t , S W t )

[0070] So the total error is:

[0071] S e t = S e a,t + S e m,t

[0072] The gyroscope is compensated for deviation through a KP controller:

[0073]

[0074] Final Posture Update:

[0075]

[0076] The preprocessing of IMU data includes ekf correction of magnetometer and error correction of accelerometer and gyroscope, which are briefly described as follows:

[0077] The magnetometer measurements can be modeled as:

[0078] B k =(I 3×3 +D) -1 (A k H k +b+ε k ),k=1,…,N

[0079] Among them, B k is the magnetometer at time t k Measurement of magnetic field, H k is the corresponding value of the geomagnetic field related to the earth's fixed coordinate system, A k is the unknown attitude matrix of the magnetometer relative to the earth-fixed coordinate system, D is an unknown diagonal matrix, where the diagonal elements represent the scale factors, the non-diagonal elements represent the non-orthogonality corrections, b is the zero bias vector, and ε k is the measurement noise vector, which is assumed to have a mean of zero and a variance of σ k Gaussian process.

[0080] The measurement equation is:

[0081] z k =||B k || 2 -||H k || 2

[0082] Then set the white noise ε according to the magnetometer technical manual k The equation of σ k , real-time estimation of D, A k , b values ​​to obtain more accurate magnetometer readings.

[0083] The accelerometer error model is:

[0084] s a m =T a K a ( s a' m +b' a +ε a )

[0085] T a is the corrected small rotation matrix, K a is the scale factor, b' a is zero bias, ε a It is white noise. The core principle is that when imu is in a relatively static state, the modulus of the specific force should be equal to the local gravitational acceleration.

[0086] We then use the least squares method to fit T a , K a , b' a To obtain more accurate motion acceleration.

[0087] The error model of the gyroscope is:

[0088] s ω m =T ω K ω ( s ω' m +b' ω +ε ω )

[0089] This error model is consistent with the accelerometer error model. The core principle is that when the imu is in a relatively static state, the angular velocity readings of the three axes should be 0. Then, the least squares method is used to fit T ω , K ω , b' ω To obtain more accurate motion acceleration.

[0090] In this embodiment, the wireless communication unit is a WiFi module, and the communication module 30 is a router. The data is sent from the WiFi module to the communication module 30 through a network transmission protocol, such as TCP protocol. Figure 3 As shown, the MCU drives the IMU and magnetometer through serial communication, such as I2C communication, and obtains their signals. The MCU drives the Wifi module through serial communication, such as SPI communication, and sends the obtained inertial data measurement values, rotation attitude and acceleration to the Wifi module.

[0091] In this embodiment, the operation and display unit is a PC, and the router is used to network the six inertial measurement modules and the PC. The router will record the physical addresses of the Wifi modules of the six inertial measurement modules and assign designated IP addresses, and then send the data to the PC synchronously through network communication protocols, such as TCP / IP protocol, to ensure the stability and real-time nature of communication, which helps to improve the user experience. The PC is configured with specific IP and DNS addresses to avoid delays caused by dynamic IP allocation; then, unnecessary background applications are closed to minimize the network delay between the PC and the router.

[0092] Figure 4 A method for predicting whole-body motion using a sparse inertial measurement unit according to an embodiment of the present invention is provided, which is also one of the methods performed by the operation display unit according to the first aspect of the present invention. The method comprises:

[0093] Step S1, multiple nodes are set at the test target, including a root node and multiple leaf nodes. In this embodiment, 6 node positions are set at the joints of the human body, the root node is set as the node behind the waist, the left and right forearms near the wrist joints, the left and right calves near the knee joints, and the node behind the head are set as leaf nodes, and an inertial measurement module is deployed at the node position;

[0094] Step S2, obtaining the node inertial motion signals measured by the six inertial measurement modules, and recording the message sequence and message timestamp of the node inertial motion signals, wherein the node inertial motion signals include information of each node of the test target;

[0095] Step S3, preprocessing the inertial motion signals of the six nodes to obtain the node inertial data measurement values, fusing the node inertial data measurement values, and obtaining the node rotation attitude and node acceleration in the geographic coordinate system;

[0096] Step S4, converting the node rotation attitude and node acceleration of the geographic coordinate system to the SMPL coordinate system, and obtaining the node rotation attitude and node acceleration of the six nodes in the SMPL coordinate system;

[0097] Step S5, through the synchronization algorithm, the node rotation posture and node acceleration of the six nodes in the SMPL coordinate system are synchronously sent to the neural network, and the positions of all leaf nodes relative to the root node are calculated;

[0098] Step S6, the position of the leaf node relative to the root node, the rotation posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the position of the whole body joints relative to the root node;

[0099] Step S7, the positions of the joints of the whole body relative to the root node, the rotational posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the rotational posture of the joints of the whole body relative to the root node;

[0100] In step S8, the node rotation postures and node accelerations of the six nodes are synchronously sent to the bidirectional recurrent neural network to estimate the probability of landing on the left and right sides. The higher value of the left and right landing probabilities is used as the effective value of the landing probability to calculate the speed of the root node.

[0101] In the further step S5, the synchronization algorithm sends the node rotation posture and node acceleration in the SMPL coordinate system with the same cached message order to the neural network synchronously. The synchronization algorithm used in one embodiment of the present invention is as follows: Figure 5 As shown, it includes: the PC machine locally allocates a thread to run a clock script, and the script increases the synchronization number sync_id by 1 every time, for example, 5 milliseconds. After confirming that all inertial measurement modules are connected to the network, the PC machine sends sync_id to the measurement module, and the inertial measurement module records this sync_id and localizes it. At this time, the localized sync_id is recorded as sync_id1. The measurement module increases the localized sync_id1 by 1 every set interval, for example, 5 milliseconds, and processes the node inertial motion signals obtained by all inertial measurement modules, obtains the node inertial data measurement values ​​and stores them in a buffer. Storing the inertial data in the buffer can effectively improve the packet loss and delay problems that may be caused by wireless communication. In this embodiment, the node inertial motion signals are fused using the heading reference system, and the node rotation posture and node acceleration in the SMPL coordinate system are obtained. The node rotation posture, node acceleration, and the inertial data measurement values ​​consistent with sync_id1 to sync_id6 are sent to the neural network for subsequent algorithm processing, and the human motion capture results are calculated, and the whole body posture is predicted by the neural network. The synchronization algorithm ensures that the data from the six measurement modules processed by the neural network are collected at the same time, which helps to improve the accuracy of motion restoration, making it easier to collect high-speed motion and restore it.

[0102] Furthermore, the training of the neural network in step S4 includes an initial calibration action, processing the action error using the SMPL coordinate system, converting the node rotation attitude and node acceleration in the geographic coordinate system to the SMPL coordinate system, and outputting the node rotation attitude and node acceleration in the SMPL coordinate system.

[0103] like Figure 6As shown, in a preferred embodiment, an extended Kalman filter EKF is used to observe the zero bias, scale factor, and non-orthogonal error of the magnetometer in real time, and the magnetometer reading is corrected. The corrected magnetometer reading and the inertial measurement unit IMU reading are fused via the attitude reference system AHRS system to obtain the rotation attitude and acceleration. In addition, a T-type action is set as the initial calibration. When the test target wears the inertial measurement module to the specified 6 nodes, the relative position and relative rotation of the inertial measurement module and the joints will affect the motion capture. The T-type action will compensate for the rotation of the measurement module relative to the joint, but it is difficult for the user to make a very standard T action, so it is processed in the neural network.

[0104] Furthermore, through coordinate system conversion, the rotational posture and acceleration of all leaf nodes are converted to the root node coordinate system, also known as the SMPL (Skinned Multi-Person Linear Model) coordinate system. This includes using an inertial navigation algorithm and an extended Kalman filter EKF to fuse the IMU information into a rotational posture; using human kinematics for posture prediction, for example, when the foot and the ground are relatively still, the spatial position of the root node behind the waist is derived based on forward kinematics using the rotational posture of the foot, calf, and thigh, and the mass of each joint is estimated according to the height and weight of the human body, and the force and torque generated during human movement are estimated; then, through a trained neural network, a bidirectional recurrent neural network (biRNN) is used in this embodiment to estimate the acceleration and rotational posture of the human end joints, including the positions of the bilateral wrists, bilateral ankles, and the top of the head relative to the root node, from the 6 IMU measurements. Then, the position of the end joint relative to the root node and the IMU information are used as input, and the biRNN network is used to infer the position of the 24 joints of the human body relative to the root joint.

[0105] Finally, the positions of the 24 joints of the human body and the IMU information are used as input, and the biRNN network is used to estimate the rotation posture of all 24 joints of the human body relative to the root node and the speed of the root node. The posture of the root node is the IMU solution posture converted to the SMPL coordinate system.

[0106] In this embodiment, physical motion constraints can be added to the speed of the root node. The accelerometers and rotational postures of the six IMUs are used as input, and the biRNN neural network is used to estimate the probability of whether the left or right side touches the ground. If the probability of touching the ground is high, the foot will not slide relative to the ground, and the speed of the root node is calculated using forward dynamics. If the probability of touching the ground is low, the RNN neural network (Recurrent Neural Network) is used to directly predict the speed of the root node from the estimated whole-body joint coordinates and the acceleration information of the six IMUs, and the speed based on the forward dynamics solution and the speed based on the RNN estimation are integrated to form the final root node geographic system speed.

[0107] In a specific embodiment, a threshold range of the landing probability is preset, for example, the threshold range is 0.5 to 0.9. If the effective value p of the landing probability is less than the minimum value of the preset threshold range, in this embodiment, when p is less than 0.5, the RNN recurrent neural network is used to predict the speed of the root node and recorded as the first speed V1; if the effective value p of the landing probability is greater than the maximum value of the preset threshold range, p is greater than 0.9, the speed of the root node is calculated using forward dynamics and recorded as the second speed V2; if the effective value of the landing probability is within the preset threshold range, the first speed V1 predicted by the RNN recurrent neural network and the second speed V2 of the root node calculated using forward dynamics are combined to calculate the third speed V3 of the root node, V3=V1*(p-0.5) / (0.9-0.5)+V2*(0.9-p) / (0.9-0.5).

[0108] The rotation posture and acceleration in the SMPL coordinate system are used as input, and the rotation posture of the 24 joints of the human body and the velocity of the root node are output. Then the data combination is sent to the 3D engine through the network transmission protocol for real-time animation display to predict the whole body posture.

[0109] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are only for illustration and not for limiting the scope of the present invention. It should be understood by those skilled in the art that modifications to the technical solutions described in the above embodiments or equivalent replacements of some of the technical features thereof may be made without departing from the scope and spirit of the present invention, and all are within the protection scope of the present invention.

Claims

1. A method for predicting whole body motion using a sparse inertial measurement unit, characterized in that: include: Step S1, setting a plurality of nodes on a test target, including a root node and a plurality of leaf nodes, and setting an inertial measurement module on the nodes; Step S2, obtaining node inertial motion signals measured by multiple inertial measurement modules, and recording the message sequence and message timestamp of the node inertial motion signals; Step S3, preprocessing the plurality of node inertial motion signals to obtain node inertial data measurement values, fusing the node inertial data measurement values, and obtaining the node rotation attitude and node acceleration in the geographic coordinate system; Step S4, converting the node rotation attitude and node acceleration of the geographic coordinate system into the SMPL coordinate system, and obtaining the node rotation attitude and node acceleration of multiple nodes in the SMPL coordinate system; Step S5, using a synchronization algorithm, synchronously sending the node rotation postures and node accelerations of multiple nodes in the SMPL coordinate system to the neural network, and calculating the positions of all leaf nodes relative to the root node; Step S6, the position of the leaf node relative to the root node, the rotation posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the position of the whole body joints relative to the root node; Step S7, the positions of the joints of the whole body relative to the root node, the rotational posture of the leaf node and the acceleration of the leaf node are synchronously sent to the neural network to calculate the rotational posture of the joints of the whole body relative to the root node; In step S8, the node rotation postures and node accelerations of multiple nodes are synchronously sent to the bidirectional recurrent neural network to estimate the probability of landing on the left and right sides, and the higher value of the left and right landing probabilities is used as the effective value of the landing probability to calculate the speed of the root node.

2. The method for predicting whole body motion using a sparse inertial measurement unit according to claim 1, wherein: The multiple nodes in step S1 are set to 6, among which the node behind the waist is set as the root node, the nodes near the wrist joints of the left and right forearms, the nodes near the knee joints of the left and right calves, and the node behind the head are set as leaf nodes.

3. The method for predicting whole body motion using a sparse inertial measurement unit according to claim 1, wherein: The synchronization algorithm of step S5 synchronously sends the node rotation posture and node acceleration in the SMPL coordinate system with consistent cached message order to the neural network.

4. The method for predicting whole body motion using a sparse inertial measurement unit according to claim 3, wherein: The training of the neural network in step S4 includes an initial calibration action, processing the action error using the SMPL coordinate system, converting the node rotation attitude and node acceleration in the geographic coordinate system to the SMPL coordinate system, and outputting the node rotation attitude and node acceleration in the SMPL coordinate system.

5. The method for predicting whole body motion using a sparse inertial measurement unit according to claim 4, wherein: In step S5, a bidirectional recurrent neural network is used to estimate the position of the leaf node relative to the root node, in step S6, a bidirectional recurrent neural network is used to estimate the position of the whole body joints relative to the root joint, and in step S7, a bidirectional recurrent neural network is used to estimate the rotation posture of the whole body joints relative to the root node.

6. The method for predicting whole body motion using a sparse inertial measurement unit according to claim 5, wherein: The speed of the root node in step S8 is calculated in the following manner: a threshold range of the landing probability is preset, and if the effective value of the landing probability is less than the minimum value of the preset threshold range, the speed of the root node is predicted using the RNN recurrent neural network and recorded as the first speed; if the effective value of the landing probability is greater than the maximum value of the preset threshold range, the speed of the root node is calculated using forward dynamics and recorded as the second speed; if the effective value of the landing probability is within the preset threshold range, the speed of the root node at this time is calculated by combining the first speed predicted by the RNN recurrent neural network and the second speed calculated using forward dynamics, and recorded as the third speed.

7. A system for predicting whole-body motion using a sparse inertial measurement unit, characterized in that: It includes an inertial measurement module, an auxiliary navigation module, a communication module and an operation display module, among which: There are multiple inertial measurement modules, which are used to be deployed at different node positions of the human body, capture node inertial motion signals, and pre-process the node inertial motion signals into node inertial data measurement values; The auxiliary navigation module is used to fuse the node inertial data measurement values ​​to obtain the joint rotation postures and joint accelerations of multiple nodes in the SMPL coordinate system; The communication module is used to obtain the joint rotation postures and joint accelerations of multiple nodes in the SMPL coordinate system, and synchronously send them to the operation display module; The operation display module executes the method for predicting whole-body motion using a sparse inertial measurement unit according to any one of claims 1 to 6 to predict whole-body posture.

8. The system for predicting whole body motion using a sparse inertial measurement unit according to claim 7, wherein: The inertial measurement module includes a microcontroller unit MCU, an inertial measurement unit IMU, a magnetometer, a wireless communication unit and a battery unit. The inertial measurement unit IMU and the magnetometer are used to capture node inertial motion signals. The microcontroller unit MCU collects the node inertial motion signals captured by the inertial measurement unit IMU and the magnetometer and performs preprocessing.

9. The system for predicting whole body motion using a sparse inertial measurement unit as claimed in claim 8, characterized in that: The microcontroller unit MCU drives the inertial measurement unit IMU and the magnetometer through serial communication, and obtains the node inertial motion signal; the microcontroller unit MCU drives the wireless communication unit through serial communication.

10. The system for predicting whole body motion using a sparse inertial measurement unit according to claim 9, wherein: There are at least six inertial measurement modules, which are respectively deployed behind the waist of the test target, behind the head, near the wrist joints of the left and right forearms, and near the knee joints of the left and right calves.

11. The system for predicting whole body motion using a sparse inertial measurement unit according to claim 10, wherein: The inertial measurement module uses an extended Kalman filter (EKF) to preprocess the node inertial motion signal to obtain the node inertial data measurement value. The auxiliary navigation module is an attitude reference system (AHRS), which fuses the node inertial motion signal through the attitude reference system (AHRS) to obtain the node rotation attitude and node acceleration.

12. The system for predicting whole body motion using a sparse inertial measurement unit according to claim 11, wherein: The communication module is a router, the wireless communication unit is a Wifi module, the operation display module is a computer, the Wifi module synchronizes the clocks of the multiple inertial measurement units IMU with the computer, and the router synchronously sends the node rotation postures and node accelerations of the multiple nodes to the operation display module through a network communication protocol.

Images